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Conclusions  will  be  used  to  plan  for  future  use  of  automated  appointment  and 
scheduling  systems  within  DoD. 
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SUMMARY  OF  FINDINGS 


In  February  1983,  an  automated  Tri-Service  Patient  Appointment 
and  Scheduling  (TRIPAS)  System  (initial  capability)  was  installed  at 
USAF  Medical  Center,  Keesler  Air  Force  Base  (Keesler),  Biloxi,  Miss¬ 
issippi.  Keesler  is  the  fourth  site  to  receive  TRIPAS  under  a  con¬ 
tract  from  the  Tri-Service  Medical  Information  Systems  (TRIMIS) 
Program  Office  (TPO). 

This  report  contains  information  on  an  evaluation  of  the  auto¬ 
mated  TRIPA.S  System  at  Keesler.  Conclusions  are  based  on  a  comparison 
of  patient  appointment  and  scheduling  operations  using  an  automated 
system  with  pre-automation  (baseline)  functions,  taking  into  account 
system  goals  and  objectives  and  user  criteria  for  patient  appointment 
and  scheduling  services.  Conclusions  drawn  about  the  functionality 
and  benefits  of  TRIPAS,  at  Keesler  and  elsewhere,  will  be  used  to  plan 
for  future  use  of  automated  appointment  and  scheduling  systems  within 
DoD. 

The  evaluation  was  designed  to  study  important  attributes  and 
features  of  the  appointment  and  scheduling  operations  that  might  be 
affected  by  the  TRIPAS  System,  including: 

•  Availability  of  appointments; 

•  Time  required  to  make,  change,  verify,  or  cancel  an 
appointment ; 

•  Waiting  times  within  clinics,  and  in  the  time  period  from 
initial  entry  into  the  medical  care  system  to  diagnosis 
and  treatment  in  the  appropriate  clinic; 

•  Availability  of  medical  records  and  X-rays  at  the  time  of 
an  outpatient  visit; 

•  The  number  of  patients  who  miss  their  appointment; 

•  Patient  and  staff  satisfaction  with  the  appointment  and 
scheduling  system; 

•  Optimization  of  patient/care  provider  encounters; 

•  Effort  required  to  make  or  change  provider  schedules;  and 
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•  Cost  per  appointment  transaction  and  per  outpatient 
visit . 

Data  that  quantified  these  features  were  collected  and  then  compared 
for  the  baseline  and  post-implementation  evaluations. 

Data  were  collected  for  the  baseline  study  from  October  £  to 
November  1,  1982.  After  collection  of  baseline  data  was  completed, 
installation  of  the  TRIPAS  System  was  closely  observed.  The  system's 
readiness  for  post-implementation  evaluation  was  documented  in  an 
interim  Implementation  Monitoring  report  (2).  Post-implementation 
data  were  collected  from  May  16-27,  1983.  In  addition,  provider 
productivity  data  were  collected  the  year  following  the  baseline  study 
for  all  of  October,  1983. 

Like  the  baseline  study,  the  post-implementation  evaluation  at 
Keesler  includes  information  gathered  from  the  Central  Appointment 
Section  (CAS),  the  Medical  Records  Department,  and  a  sample  of  repre¬ 
sentative  outpatient  clinics.  Five  clinics  were  included  in  the 
study:  Primary  Care,  Internal  Medicine,  General  Surgery,  and  Ortho¬ 
pedics,  plus  the  Dental  Clinic  for  which  only  qualitative  data  were 
collected.  These  clinics  were  selected  on  the  basis  of  their  sched¬ 
uling  methods  and  the  volume  of  patients  seen,  especially  active-duty 
personnel.  Two  of  the  clinics  in  this  sample  (Internal  Medicine, 
Primary  Care)  used  CAS  for  making  patients'  appointments;  the  others 
booked  appointments  using  their  own  appointment  and  scheduling  clerks. 
Results 

Table  S-l  compares  the  results  of  the  baseline  evaluation  to  the 
results  of  the  post-implementation  evaluation.  The  table  and  the 
subsequent  discussion  are  organized  to  correspond  to  the  process  a 
patient  goes  through  in  receiving  routine  outpatient  care:  requesting 
an  appointment;  being  scheduled  for  an  appointment;  having  a  medical 
record  located  and  sent  to  the  appropriate  clinic;  and  being  seen  by  a 
care  provider  at  the  appointed  time.  Keesler  staff  were  initially 
asked  their  perception  of  minimum,  average  and  maximum  expected  goals 
for  some  areas  that  might  be  improved  by  the  TRIPAS  System.  Table  S-2 
outlines  these  goals. 
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TABLE  S-l 


SUMMARY  OF  BASELINE  AND  POST-IMPLEMENTATION 
EVALUATION  RESULTS,  TRIPAS  EVALUATION  AT 
USAF  MEDICAL  CENTER,  KEESLER  AIR  FORCE  BASE 


Evaluation 
Base  Post 


Average  waiting  time  (sec)  for 


CAS  caller,  when  queue  existed 

117 

76 

-41 

sec 

Average  number  of  CAS  callers  in 

queue,  when  queue  existed 

4.8 

4.7 

-0. 

1  callers 

Percentage  of  time  CAS  queue  existed 

Percentage  of  patients'  attempts  to 
call  before  getting  an  appointment: 

36 

23 

13 

percentage 

points 

1  attempt 

44 

38 

-6 

percentage 

points 

2  attempts 

16 

19 

+3 

percentage  points 

3  attempts 

15 

19 

+4 

percentage 

points 

More  than  3  attempts 

Percentage  of  respondents  satisfied* 
with  telephone  system: 

21 

19 

-2 

percentage 

points 

Patients 

57 

61 

+4 

percentage 

points 

Providers 

27 

40 

+  13 

percentage 

points 

Support  staff 

65 

74 

+9 

percentage 

points 

Scheduling 


3 ointment 


Average  time  (sec)  required  to  book 
an  appointment: 

In  CASb 

In  clinics  -  Orthopedics 

-  General  Surgery 

Number  of  appointment  calls  per 
clerk  per  hour  in  CAS 


82 

78 

-4  sec 

. .. . 

201 

167 

-34  sec 

j  y\  '  r*  r*  s  1  Of!  V  oT 

130 

192 

+62  sec 

r  TAB 

r  i  '  ^ 

13 

10 

-4  calls 

'  .  *.  »  i  1 i  -■  • 
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TABLE  S-l  (continued) 


Evaluation 

Impact  Area  Base  Post  Change 

Scheduling  an  Appointment  (continued) 

Percentage  of  respondents  indicat¬ 
ing  that  patient  appointment  and 
scheduling  system  is  important2  to 
the  quality  of  care  at  the  facility: 


Providers 

92 

96 

+4 

percentage 

points 

Support  staff 

93 

94 

+  1 

percentage 

points 

3 

Percentage  of  respondents  satisfied 
with  overall  quality  of  appointment 
and  scheduling  service  or  with 
appointment  system  in  general: 


Patients 

65 

72 

+7 

percentage 

points 

Providers0 

49 

63 

+  14 

percentage 

points 

Support  staff 

75 

88 

+  13 

percentage 

points 

Percentage  of  respondents  with  same 
day  multiple  clinic  appointments 

6.3 

9.2 

+2.9  percentage  point 

Locating  Medical  Records 

Percentage  of  scheduled  patients  who 
have  a  medical  record  available  at 
time  of  appointment 

85 

87 

+2 

percentage 

points 

g 

Percentage  of  respondents  satisfied 
with  availability  of  medical  records: 

Patients 

86 

87 

+  1 

percentage 

points 

Providers 

69 

72 

+3 

percentage 

points 

Receiving  Care  at  the  Appointed  Time 

Percentage  of  patients  having  care 
provider  contact  within  15  minutes 
of  their  appointment 

40 

44 

+4 

percentage 

points 

Average  waiting  time  (min)  for 
scheduled  patients 

29 

26 

-3 

min 
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TABLE  S-l  (continued) 


Impact  Area 


Evaluation 
Base  Post 


Change 


Receiving  Care  at  the  Appointed  Time  (continued) 

Percentage  of  respondents  satisfied 
with  waiting  times  in  the  clinic: 


Patients 

Providers 

Percentage  of  "no-show"  patients  in 
all  four  clinics 

Number  of  patients  seen  per  provider 
hour  (sample  of  8  providers) 

Number  of  patients  seen  per  provider 
hour — all  providers 


63 

64 

+  1 

percentage 

points 

72 

80 

+8 

percentage 

points 

6.0 

5.8 

-0.2 

percentage 

points 

3.04 

3.27 

+0.23 

percentage 

points 

1.84 

2.15 

+0.31 

percentage 

points 

43 

74 

+3 1 C 

levels  of  satisfaction  and  importance  are  defined  as  including  "very"  and 
"somewhat"  positive  responses  to  a  question. 

3See  Table  S-2  for  performance  goals  established  for  this  area. 


KEY:  Base  =  baseline  (manual  system)  study. 

Post  =  post-implementation  (automated  system)  study. 
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TABLE  S-2 


PERFORMANCE  GOALS  ESTABLISHED  FOR  DESIGNATED  IMPACT  AREAS, 
TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


_ Performance  Goals _ 

Impact  Area  Minimum  Average  Maximum 

Time  required  to  book  an  appoint¬ 
ment  in  CAS  144  sec*  240  sec  420  sec 

Percentage  of  respondents  satisfied 
with  overall  quality  of  appointment 
and  scheduling  service  or  with 
appointment  system  in  general: 

Providers 
Support  staff 

Percentage  of  scheduled  patients  who 
have  a  medical  record  available  at 

time  of  appointment  75%  89%  97T* 


*Best  value  likely  to  be  achieved. 


56%  75%  87%* 

56%  81%  91%* 
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With  automation,  the  average  waiting  time  to  answer  a  call  placed 
in  the  queue  to  CAS  decreased  by  41  seconds.  The  percentage  of  time 
that  a  queue  existed  declined;  however,  the  average  number  of  callers 
in  the  queue  did  not  change,  and  the  number  of  patients  who  reported 
they  had  successfully  booked  an  appointment  on  the  first  call  de¬ 
creased  by  6%.  What  did  improve  was  patient,  care  provider,  and 
appointment  and  scheduling  (support)  staff  satisfaction  with  the 
telephone  appointment  system.  TRIPAS  also  reduced  the  waiting  time  in 
the  CAS  telephone  queue  and  the  average  number  of  calls  in  the  queue. 

The  average  time  to  book  an  appointment  in  CAS  dropped  from  82 
seconds  before  automation  to  78  seconds  after  automation  —  an  im¬ 
provement  of  5%.  CAS  clerks  booked  appointments  at  a  rate  averaging 
about  10  appointment  calls  per  hour.  In  the  sample  clinics,  booking 
time  improved  in  Orthopedics  but  deteriorated  in  General  Surgery. 

Most  care  providers  (96%)  and  support  staff  (94%)  continued  to 
consider  the  patient  appointment  and  scheduling  function  important  to 
the  quality  of  care  at  Keesler.  The  level  of  satisfaction  of  pa¬ 
tients,  care  providers,  and  support  staff  with  the  overall  quality  of 
the  appointment  and  scheduling  service  had  improved  by  7-14%  at  the 
time  of  the  post-implementation  study. 

Eighty-seven  percent  of  patients  appointed  under  the  automated 
system  had  a  medical  record  available  at  the  time  of  their  appoint¬ 
ment,  compared  with  85%  in  the  manual  system.  There  was  little 
variation  between  systems  in  care  provider  and  patient  satisfaction 
with  availability  of  medical  records. 

Patients  waited,  on  average,  29.3  minutes  before  seeing  a  care 
provider  with  the  manual  appointment  system,  and  26.4  minutes  with  the 
automated  system.  Forty  percent  of  appointed  patients  saw  a  care 
provider  within  15  minutes  of  the  time  of  their  appointment  in  the 
manual  system  compared  with  44%  in  the  automated  system.  The  average 
no-show  rate  for  patients  with  appointments  remained  constant  at  about 
6%  for  the  four  clinics  sampled.  However,  there  was  variation  in  t  ne 
magnitude  and  direction  of  the  change  in  no-show  rates  aircrg  the 
clinics . 
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Based  on  a  design  that  would  track  continuity  of  patient  care 
from  primary  entry  through  specialized  clinic  responses,  staff  and 
patients  at  Keesler  were  selected  and  interviewed.  In  general, 
interviewers  felt  that  TPIPAS  had  improved  the  appointment  and  sched¬ 
uling  process.  The  Commander  of  the  Center  stated: 

All  around  very  helpful.  Generally  a  big  plus.  It's  helped 
with  morale.  It  makes  me  smile  to  see  very  young  techs 
working  the  system  and  becoming  proficient  in  a  short  period 
of  time.  Its  reports  have  brought  me  closer  to  the  physi¬ 
cian  staff. 

Interviews  with  patients  referred  from  Primary  Care  to  a  special¬ 
ty  clinic  (General  Surgery  and  Orthopedics)  revealed  that  once  seen  in 
Primary  Care,  active-duty  patients  referred  to  General  Surgery  waited 
an  average  of  3  days  for  an  appointment  with  manual  and  automated 
systems,  while  non-active-duty  personnel  waited  an  average  of  5  days 
(manual)  and  6  days  (automated)  for  an  appointment. 

The  cost  of  an  appointment  and  scheduling  transaction  within  CAS, 
based  on  FY  1983  costs  and  workload,  increased  from  43  cents  to  74 
cents  with  the  automated  system. 

Goals  and  Objectives 

Despite  the  fact  that,  at  the  time  of  the  post-implementation 
study,  there  were  only  3  months'  worth  of  accumulated  experience  with 
the  system  in  operation  at  Keesler,  the  comparison  of  the  system 
performance  to  goals  and  objectives  indicates  that  at  that  time,  six 
of  the  10  objectives  had  been  met  and  two  more  objectives  had  been 
partially  met.  The  system  had  reduced  the  time  to  make  patient 
appointments,  reduced  clinic  waiting  time,  reduced  appointment  cleri¬ 
cal  work,  increased  utilization  of  providers,  reduced  appointment 
errors,  and  improved  interfacility  communication.  There  were  indi¬ 
cations  that  the  system  was  going  to  provide  information  that  would 
enhance  decision  making  and  would  help  in  establishing  a  data  bank  of 
information  to  use  in  facilities  management. 

There  was  a  2.9%  increase  in  the  number  of  patients  who  were  able 
to  book  multiple,  same-day  appointments.  A  reduction  in  no-shows  had 
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not  been  experienced  after  three  months.  Both  these  impacts  require 
that  patients  become  familiar  with  the  system,  that  they  are  knowl¬ 
edgeable  about  its  capabilities,  and  that  they  change  their  behavior. 
As  patients  learn  that  they  will  be  seen  at  their  appointed  time  and 
that  they  are  able  to  make  multiple,  same-day  appointments,  it  is 
anticipated  that  the  number  of  such  appointments  booked  will  increase 
further.  A  reduction  in  no-shows  would  be  possible  by  educating  and 
reminding  patients  who  do  not  show  up  for  their  annointments ,  whose 
names  would  be  available  from  the  system,  as  tr  ,tu  importance  of 
showing  up  at  the  appointed  time. 
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I.  INTRODUCTION 


A.  EVALUATION  OBJECTIVES 

In  February  1983,  an  automated  Tri-Service  Patient  Appointment 
and  Scheduling  (TRIPAS)  System  (initial  capability)  replaced  the 
manual  appointment  system  at  the  USAF  Medical  Center,  Keesler  Air 
Force  Base  (Keesler),  Biloxi,  Mississippi.  This  installation  was 
carried  out  as  part  of  a  contract  from  the  Tri-Service  Medical  Infor¬ 
mation  Systems  (TRIMIS)  Program  Office  (TPO) .  The  experience  with  the 
implementation  of  this  system  is  being  evaluated  in  order  to  provide 
TPO  with  information  concerning  the  costs  and  benefits  of  providing 
automated  support  to  patient  appointment  and  scheduling  in  Military 
Treatment  Facilities  (MTFs).  The  first  live  day  of  operation  for 
TRIPAS  at  Keesler  was  February  28,  1983.  This  report  presents  results 
of  the  post-implementation  assessment  of  the  automated  TRIPAS  System 
at  Keesler,  and  compares  these  results  with  previously  reported 
baseline  findings 

The  purpose  of  this  overall  evaluation  effort,  both  baseline  and 
post-implementation,  is  to  assess: 

•  performance  and  impact  of  the  TRIPAS  System;  and 

•  desirable  improvements  in  the  TRIPAS  System. 

The  impact  of  the  TRIPAS  System  was  measured  by  documenting 
changes  in  the  performance  of  the  appointment  and  scheduling  system 
between  the  baseline  (manual)  and  post-implementation  (automated) 
periods.  The  comparison  was  accomplished  both  by  measurement  of 
performance  indicators  and  by  qualitative  assessment  of  patient  and 
staff  perceptions  and  acceptance  of  the  system.  Both  quantitative  and 
qualitative  information  were  used  to  determine  to  what  extent  the 
TRIPAS  System  has  met  or  exceeded  user  expectations  and  goals. 

B.  SETTING 

USAF  Medical  Center  Keesler  is  designated  as  an  area  medical  and 
consultant  center,  providing  health  care  to  beneficiaries  residing 
throughout  the  southeastern  region  of  the  United  States.  To  accom¬ 
plish  this  mission,  the  medical  center  currently  operates  a  325-bed 
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medical  treatment  facility,  a  50-bed  aeromedical  staging  flight,  a 
78-room  dental  treatment  facility,  and  40  outpatient  clinics.  The 
medical  center  includes  approximately  325  officers,  800  enlisted 
personnel,  and  230  civilians.  In  addition,  an  average  of  105  officers 
and  30  enlisted  personnel  are  assigned  to  Keesler  for  specialized 
training. 

Representative  average  monthly  workload  volumes  at  this  facility 
include  950  inpatient  admissions,  34,300  outpatient  visits,  and  2,250 
dental  visits.  The  medical  center's  ancillary  departments  also 
experience  heavy  monthly  workload:  108,000  clinical  laboratory 
procedures,  9,000  dental  laboratory  procedures,  40,000  radiology 
procedures,  and  55,000  outpatient  prescriptions. 

C.  TRIPAS  SYSTEM  AT  KEESLER  AFB 

The  Tri-Service  Patient  Appointment  and  Scheduling  System  pro¬ 
vides  state-of-the-art  automated  support  to  clinic  appointment  sched¬ 
uling.  The  functions  supported  include  appointments,  schedules, 
registrations,  medical  records,  and  report  generation. 

TRIPAS  is  an  adaptation  of  an  automated  patient  appointment 
system,  the  software  for  which  was  developed  by  Medical  Data  Corpo¬ 
ration  and  acquired  by  HBO  of  Atlanta,  Georgia.  Martin  Marietta,  of 
Bethesda,  Maryland,  is  the  prime  contractor,  with  HBO  the  subcontrac¬ 
tor  responsible  for  developing  and  supporting  system  software.  TRIPAS 
software  is  written  in  standard  MUMPS. 

TRIPAS  is  an  adaptation  of  an  automated  patient  appointment  and 
scheduling  system,  the  software  for  which  was  developed  by  Medical 
Data  Corporation,  currently  wholly  owned  by  HBO,  Atlanta,  GA.  Martin 
Marietta,  Bethesda,  MD,  is  the  Prime  contractor,  with  HBO  the  subcon¬ 
tractor  responsible  for  developing  and  supporting  system  software. 
TRIPAS  software  is  written  in  standard  MUMPS.  There  are  three  sizes 
of  Data  General  CPU  hardware  configuration:  large  (MV8000) ;  medium 
(S240) ;  small  (S-140). 

Keesler  received  the  large  configuration,  which  includes: 
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Equipment  Number  (estimated) 

Central  Processing  Unit  (CPU)  1 

Line  Printer  1 

Character  Printer  20 

Cathode  Ray  Tube  (CRT)  66 

Automated  Embossing  Machine  2 

Seven  CRTs  are  located  in  the  Central  Appointments  Section  (CAS)  and 
most  of  the  outpatient  clinics  will  have  at  least  one  terminal.  The 
CPU  is  housed  in  a  specially  secured  and  air  conditioned  computer  room 
that  also  houses  the  CPUs  for  the  TRIPAD  and  TRIPHARM  Systems  and  will 
house  the  TRILAB  and  CAPOC  System's  CPUs. 

D.  ORGANIZATION  OF  THE  REPORT 

The  remainder  of  this  report  is  divided  into  three  chapters. 
Chapter  II  describes  the  methodology  for  the  evaluation. 

Chapter  III  presents  findings,  organized  into  sections  that 
correspond  to  four  steps  in  the  process  whereby  a  patient  seeks  and 
then  receives  outpatient  care: 

•  Requesting  an  appointment; 

•  Being  scheduled  for  an  appointment; 

•  Having  a  medical  record  located  and  sent  to  the  appropri¬ 
ate  clinic;  and 

•  Receiving  care  at  the  appointed  time. 

For  each  step  of  the  process,  activities  involved  are  described,  the 
results  of  the  baseline  evaluation  presented,  and  the  implications  of 
these  results  discussed.  The  last  section  in  Chapter  III  presents  an 
analysis  of  system  costs  and  potential  savings. 

Chapter  IV  examines  the  goals  established  for  the  TRIPAS  System 
and  compares  the  benefits  derived  from  the  system,  as  outlined  in  this 
assessment,  to  those  goals. 

Appendix  A  contains  the  composite  data  compiled  from  the  various 
questionnaires  administered  during  the  post-implementation  period. 

In  both  baseline  and  post-implementation  periods,  only  patient 
and  provider  satisfaction  were  assessed  for  the  Dental  Clinic. 
Although  our  baseline  evaluation  reported  data  for  all  clinics 
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including  Dental,  the  Dental  Clinic  operations  are  somewhat  unique. 
Therefore,  the  assessment  is  more  representative  of  the  typical  clinic 
experience  if  data  from  the  Dental  Clinic  are  analyzed  and  documented 
separately.  For  that  reason,  a  discussion  of  the  Dental  Clinic  is 
included  as  Appendix  B. 
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II.  METHODOLOGY 


A.  APPROACH 

The  overall  approach  to  evaluating  the  TRIPAS  System  was  a 
"before  and  after"  study  of  the  performance  (effectiveness)  and  cost 
of  outpatient  appointment  and  scheduling  services  at  Keesler.  The 
basic  performance  indicators  selected  for  study  and  evaluation  tech¬ 
niques  were  the  same  as  those  employed  previously  in  the  focused 

evaluations  at  Brooke  Army  Medical  Center  and  Ehrling  Bergquist  USAF 
(3) 

Regional  Hospital 

In  1978,  the  TRIMIS  Medical  Review  Group  stated  the  following 

(4) 

goals  and  objectives  for  TRIPAS  : 

•  Reduced  time  to  make  a  patient  appointment; 

•  Reduced  clinic  waiting  time; 

•  Reduced  trips  to  receive  care  in  multiple  clinics; 

•  Reduced  appointment  clerical  work; 

•  Increased  utilization  by  better  matching  patients, 
providers,  facilities; 

•  Reduced  "no-shows"; 

•  Reduced  appointment  transcription  and  processing  errors; 

•  Improved  Interfacility  communication; 

•  Enhanced  decision  making;  and 

•  Establishment  of  a  data  bank. 

This  evaluation  was  designed  to  assess  —  quantitatively,  quali¬ 
tatively,  or  by  both  means  —  the  ability  of  TRIPAS  to  meet  each  of 
these  objectives. 

In  order  to  formulate  plans  for  evaluating  the  impact  of  TRIPAS 
at  Keesler,  interviews  were  conducted  with  Keesler  staff  during  a 
2-day  site  visit  on  September  9  and  10,  1982.  In  addition  to  evaluat¬ 
ing  the  operation  of  the  Central  Appointment  Section  (CAS),  evaluation 
staff  selected  five  outpatient  clinics  for  study:  two  clinics  using 
CAS  to  make  appointments  and  three  that  book  their  own  appointments. 
In  one  of  the  latter,  the  Dental  Clinic,  only  provider  and  patient 
levels  of  satisfaction  were  assessed. 
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Table  1  displays  clinic  workload,  number  of  assigned  providers, 
and  appointment  process  in  each  of  the  clinics  that  were  initially 
selected  for  study  in  this  evaluation.  The  study  clinics  were  select¬ 
ed  on  the  basis  of  their  scheduling  methods  and  the  volume  of  patients 
seen,  especially  active-duty  personnel.  Though  the  evaluation  at 
Keesler  involved  a  sample  of  all  patients  who  use  the  clinics  under 
study,  particular  attention  was  given  to  the  impact  of  TRIPAS  cn 
appointment  and  scheduling  activities  for  active-duty  personnel. 

Table  2  summarizes  the  impact  areas  and  evaluation  techniques 
selected  for  study  in  this  assessment.  These  areas  and  techniques 
were  originally  outlined  in  the  Evaluation  Plan  (5). 

Baseline  data  were  collected  in  October  1982,  in  accordance  with 
the  data  collection  procedures  outlined  in  Table  2.  Results  of  the 
baseline  evaluation  were  presented  in  February  1983  (1). 

After  the  system  was  installed  and  its  operation  had  stabilized, 
implementation  monitoring  was  conducted.  Working  with  TPO,  19  crite¬ 
ria  and  their  corresponding  measures  were  developed  to  determine 
progress  of  implementation  and  system  readiness  for  evaluation. 
Implementation  monitoring  was  conducted  during  a  2-day  site  visit  in 
early  April  1983.  Data  were  collected  in  the  five  sample  clinics, 
CAS,  and  Medical  Records.  TRIPAS  System  performance  criteria  and  the 
results  of  implementation  monitoring  are  displayed  in  Appendix  C.  The 

TRIPAS  System  met  13  out  of  19  criteria  that  were  established  for 

(2) 

determining  readiness  for  evaluation  ,  and  a  decision  was  made  to 
collect  post-implementation  data  in  May  to  avoid  the  inevitable 
disruption  in  staff  and  schedules  that  occurs  over  the  summer.  System 
operation  had  stabilized  sufficiently  for  evaluation,  even  considering 
its  short  operating  time  (3  months)  at  the  time  of  post- implement at ion 
data  collection. 

B.  ASSUMPTIONS  AMD  CONSTRAINTS 

The  following  assumptions  were  made  in  conducting  this  eval¬ 
uation  : 

•  The  perceptions  of  the  individuals  surveyed  accurately 
assess  the  functioning  of  the  system.  They  have  had 
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TABLE  1 


CHARACTERISTICS  OF  CLINICS  SELECTED  FOR  INCLUSION 
IN  THE  TRIPAS  EVALUATION  AT  USAF  MEDICAL 
CENTER,  KEESLER  AFB 


Clinic 

Annual 

Outpatient 

Visits2 

Number  of 

Assigned 

Providers* 

Appointment 

Process 

Dental 

26,400 

22 

Books  own  appointments 

General  Surgery 

29,100 

25 

Books  own  appointments 

Internal  Medicine 

8,000 

45 

Uses  CAS 

Orthopedics 

13,200 

5 

Books  ox^n  appointments 

Primary  Care 

51,000 

15 

Uses  CAS 

d 

June  16,  1982  survey  of  clinic  workload  statistics  conducted  by 
Keesler  hospital  services. 
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enough  experience  with  Keesler  operations  and  the  patient 
appointment  and  scheduling  system  to  have  informed 
opinions  concerning  system  performance. 

•  The  impact  of  the  observers  on  operations  of  CAS  and  the 
clinics  was  insignificant. 

•  The  differences  observed  in  MTF  operations  between  the 
baseline  and  post-implementation  periods  were  due  to  the 
TRIPAS  System  unless  evidence  existed  to  the  contrary. 

If  staffing  workload,  appointment  procedures,  and  patient 
mix  were  not  equivalent  during  the  two  periods,  the  data 
analyses  were  assumed  capable  of  adjustment  for  the 
differences  or  were  at  least  useful  in  explaining  the 
causes  of  these  differences. 

•  The  impact  areas  studied  accounted  for  the  most  signifi¬ 
cant  and  meaningful  changes  occurring  as  a  result  of 
implementing  the  TRIPAS  System  at  Keesler. 

•  Sufficient  data  were  collected  and  Keesler  operations 
were  observed  for  a  sufficient  amount  of  time  during  a 
reasonably  stable  time  period  to  accurately  reflect 
normal  operations. 

•  The  clinics  selected  for  study  were  typical  of  the 
methods  of  operation  throughout  Keesler  Medical  Center. 

These  assumptions  appear  reasonable  for  an  evaluation  of  this  type 
(before-after)  and  duration  (10-day  data  collection  period). 

Three  major  constraints  have  been  identified  in  the  conduct  of 
this  study: 

•  In  order  to  keep  evaluation  costs  reasonable,  data  were 
not  collected  at  a  "control"  MTF,  e.g.,  one  that  is  not 
implementing  a  TRIPAS  system;  thus,  the  evaluation  is  not 
a  classical,  more  definitive  "before-after"  case  control 
study. 

•  Beginning  in  May  of  every  year,  there  is  turnover  of 
physicians  and  other  health  care  providers  at  Keesler; 
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thus,  some  baseline  study  participants  were  lost  to 
follow-up  in  the  post-implementation  survey. 

•  Workloads  were  not  equal  in  the  baseline  and  pcst- 
implementation  periods: 

-  After  the  baseline  period  one  clinic  (ENT)  switched 
from  booking  its  own  appointments  to  having  CAS  book 
them. 

-  Questionnaire  results  revealed  that  up  to  one  third  of 
the  patients  in  clinics  that  booked  through  CAS  were 
being  appointed  instead  by  clinic  personnel  using  the 
automated  system;  this  phenomenon  may  have  reduced  CAS 
workload  in  the  post-implementation  period. 

Despite  these  complications,  execution  of  the  study  design  appears  to 
have  fulfilled  the  evaluation  objectives  of  this  project. 

C.  DATA  COLLECTION 

Baseline  data  collection  began  on  October  4,  1982,  and  continued 
to  the  end  of  the  month.  Implementation  monitoring  conducted  during 
April  1983  indicated  that  the  TRIPAS  System  was  ready  for  evaluation. 
Post-implementation  data  collection  was  conducted  from  May  16  to  May 
27,  1983.  Previous  staff  experience  on  site  contributed  to  efficient 
data  collection  during  the  post-implementation  period. 

Beginning  on  May  16,  1983,  post-implementation  data  were  collect¬ 
ed  in  the  five  sample  clinics  and  CAS  for  10  days.  All  data  were 
recorded  by  evaluation  team  members  or  hired  and  closely  supervised 
data  collectors,  with  the  exception  of  cost  data  and  reports  concern¬ 
ing  availability  of  clinic  appointments,  which  were  obtained  from 
TRIMIS  records  and  existing  hospital  reports.  Prior  to  implementation 
of  data  collection,  on-site  data  collectors  (primarily  off-duty  Red 
Cross  volunteers  who  were  familiar  with  Keesler  operations  and  had 
collected  baseline  data)  were  briefed  by  staff  from  National  Capitol 
Systems,  Inc.  (NCSI)  on  data  collection  procedures,  given  a  demon¬ 
stration,  and  then  checked  for  proficiency  and  accuracy  as  they 
performed  the  data  collection  tasks. 
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The  number  of  observations  made  during  both  baseline  and  post¬ 
implementation  periods  is  included  as  Table  3. 

Data  concerning  the  availability  of  appointments  and  workload 
were  collected  at  CAS  and  in  the  two  clinics  that  book  their  own 
appointments  (Orthopedic-  and  General  Surgery) .  Clerks  recorded  all 
incoming  calls  by  maintaining  logs  by  type  of  call:  appointment 
booked,  appointment  not  available,  appointment  cancelled,  appointment 
verified,  or  information  given.  Data  collectors  also  maintained 
records  of  kept  and  unkept  appointments,  and  walk-in  patients,  in  the 
four  sample  clinics,  and  tracked  sample  patients  through  the  Primary 
Care  Clinic  screening  process  to  diagnosis  and  treatment  in  another 
appropriate  clinic.  This  tracking  facilitated  the  assessment  of  the 
availability  of  appointments  from  a  continuity  of  care  perspective. 

An  automated  system  in  use  at  Keesler  records  the  amount  of  time 
patients  calling  CAS  wait  before  the  telephone  is  answered.  This 
information  was  provided  at  regular  intervals  throughout  the  day. 
This  system  eliminated  the  need  for  data  collectors  to  make  on-base 
calls  to  gather  information  on  caller  waiting  times. 

In  order  to  measure  the  amount  of  time  spent  in  the  appointment 
and  scheduling  process,  direct  observation  of  hospital  activities  was 
performed.  Timed  observations  were  made  to  record  patient  waiting 
times  in  the  clinics.  The  time  required  to  reach  a  CAS  clerk  on  the 
telephone  and  the  amount  of  time  required  to  book  an  appointment  were 
measured . 

Two  physicians  per  clinic  were  interviewed  in  order  to  gather 
information  regarding  physician  productivity  as  well  as  the  provider 
perspective  on  the  availability  of  appointments. 

To  calculate  physician  productivity,  data  were  collected  on  the 
average  number  of  patients  physicians  saw  per  working  hour  during  the 
post-implementation  study  and  this  figure  was  compared  with  the 
baseline  figure.  Because  some  physicians  were  not  available  in  both 
periods,  data  from  a  sample  of  eight  full-time  non-resident  physicians 
was  initially  available  for  analysis.  These  data  showed  that  the 
average  number  of  patients  seen  per  hour  by  these  eight  physicians 
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NUMBER  OF  BASELINE  AND  POST- IMPLEMENTATION  OBSERVATIONS  MADE 
TR1PAS  EVALUATION,  USAF  MEDTCAI.  CENTER,  KEESLER  AFB 
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KEY:  CAS  =  Central  Appointment  Section;  IM  =  Internal  Medicine;  PC  =  Primary  Care;  OS  =  General  Surperv;  0 

Orthopedics ;  D  =  Dental. 


increased  from  3.04  in  the  baseline  to  3.27  at  the  time  of  the  post¬ 
implementation  study.  While  the  observed  change  was  relatively  small 
(.23  patients  per  hour  per  physician  increase),  the  resulting  value  of 
this  benefit  is  extremely  sensitive  to  small  variations  and  could 
produce  very  large  benefits  for  TRIPAS. 

Because  this  preliminary  estimate  was  based  on  such  a  small 
sample  of  physicians,  additional  data  were  obtained.  Data  reflecting 
all  outpatient  visits  and  physician  hours  spent  in  the  23  clinics  at 
Keesler  AFB  were  obtained  for  both  October  1982  and  October  1983.  The 
number  of  visits  per  hour  was  calculated  for  each  clinic  pre  and  post 
the  implementation  of  the  PAS  system.  A  two-tailed  t-test  was  per¬ 
formed  on  the  data.  The  difference  was  found  to  be  statistically 
significant  at  the  .01  level.  The  mean  number  of  visits  per  hour  in 
all  Keesler  Clinics  by  all  providers  in  October  was  1.84  before  PAS 
and  2.15  after  PAS,  resulting  in  an  average  increase  of  .31  patients 
per  physician  hour.  (See  Appendix  D  for  detailed  data  on  provider 
productivity  and  related  statistical  analysis.) 

Keesler  financial  records  were  reviewed  in  order  to  determine  the 
number  of  scheduling  personnel  and  their  salaries,  the  costs  of 
equipment  and  supplies,  and  the  costs  of  base  overhead  support.  A 
review  of  TRIPAS  System  procurement  costs  was  performed  to  calculate 
post-implementation  systems  costs.  This  provided  cost  data  from  which 
the  cost  per  unit  of  workload  (cost  per  appointment,  cost  per  appoint¬ 
ment  transaction)  was  determined. 

Surveys  were  used  to  assist  in  establishing  performance  criteria 
and  to  measure  satisfaction  with  current  services.  Patients,  provi¬ 
ders,  and  scheduling  staff  in  all  five  study  clinics  were  surveyed  by 
use  of  a  self-administered  questionnaire.  These  questionnaires 
contained  questions  concerning  satisfaction  with  various  aspects  of 
the  appointment  and  scheduling  system  and  concerning  performance  goals 
for  the  system,  and  open-ended  questions  about  system  improvement. 

Supervisors  and  administrators  in  the  five  study  clinics,  CAS, 
Medical  Records,  Patient  Affairs,  and  Administration  were  interviewed 
regarding  their  satisfaction  with  the  automated  patient  appointnent 
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and  scheduling  system  and  their  suggestions  for  improvements.  They 
were  asked  in  what  ways  had  TRIPAS  improved  clinic  efficiency  and 
their  clinical  effectiveness. 

Data  collection  activities  were  completed  in  May  1983.  Data 
coding  and  analyses  were  completed  in  June  1983  and  this  report  was 
prepared  in  August  1983. 

D.  ANALYSIS 

Analysis  of  the  data  was  of  two  general  types: 

•  Comparisons  of  post-implementation  results  with  baseline 
results  for  the  same  impact;  and 

•  Comparisons  of  baseline  and  post-implementation  results 
with  performance  goals  set  by  Keesler  staff,  and  with 
system  goals  and  objectives  set  by  the  Medical  Review 
Group. 

For  both  types  of  analysis  quantified  results  are  presented  side  by 
side  and  improvement  or  deterioration  in  performance  discussed. 
Results  were  analyzed  for  each  clinic  and  CAS  as  well  as  aggregated 
for  all  clinics.  In  general,  impacts  are  clustered  in  one  of  five 
sections  of  this  report: 

•  Requesting  an  appointment; 

•  Being  scheduled  for  an  appointment; 

•  Having  the  medical  records  located  and  sent  to  the 
appropriate  clinic; 

•  Receiving  care  at  the  appointed  time;  or 

•  Cost  of  appointment  and  scheduling  operations. 

Where  the  differences  between  observation  periods  required  inter¬ 
pretation,  statistical  tests  were  performed  (chi-square,  t-tests)  to 
assess  the  significance  of  differences  between  performance  periods. 
Differences  in  average  clinic  waiting  time  to  see  a  care  provider,  for 
example,  were  considered  significant  if  the  probability  of  the  dif¬ 
ference  occurring  by  chance  was  less  than  one  in  twenty  (p  <  0.05). 
Sufficient  samples  were  collected  in  both  observation  periods  to 
detect  differences  at  this  significance  level.  The  data  compared  for 
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quantified  impacts,  in  general,  involved  stratified  means  and  standard 
deviations. 

Questionnaire  data  were  keypunched,  edited,  and  analyzed  using  a 
standard  statistical  computer  program.  A  Likert  Scale  was  used  to 
record  responses  concerning  importance  of  and  satisfaction  with  the 
appointment  system  into  mean  ratings. 

The  performance  goals  defined  by  Keesler  staff  were  set  for  three 
levels : 

•  Minimum  acceptable. 

•  Average  expected. 

•  Maximum  attainable. 

Comparison  of  baseline  and  post-implementation  results  with  these 
levels  of  anticipated  performance  (as  determined  in  a  provider  survey) 
allowed  an  assessment  of  how  well  the  TRIPAS  System  met  user  expecta¬ 
tions.  These  analyses  provided  a  second  method  of  interpreting  the 
significance  of  changes  in  performance  between  the  manual  and  auto¬ 
mated  system.  Performance  goals  were  established  for  about  one-third 
of  the  TRIPAS  impacts  that  were  assessed. 

Qualitative  data  obtained  from  the  questionnaires  generally  fell 
into  five  categories:  "very"  or  "somewhat"  positive,  neutral,  and 
somewhat  or  very  negative.  For  this  evaluation,  the  very  and  somewhat 
positive  responses  have  been  combined  into  one  positive  figure  in¬ 
dicating  an  acceptable  level  of  satisfaction  or  importance. 

In  general,  baseline  results  were  compared  with  post-implemen¬ 
tation  results.  In  order  to  perform  this  comparison,  similar  ana¬ 
lytical  methods  have  been  applied  to  the  data  for  both  periods.  In 

some  cases  this  has  required  recalculation  of  baseline  results  in 

order  to  ensure  that  the  data  are  comparable.  Baseline  results  for 

some  variables  may  therefore  vary  somewhat  from  those  same  variables 
as  reported  in  the  baseline  evaluation  (1). 
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V 


III.  RESULTS 


This  chapter  presents  the  results  of  the  post-implementation 
(automated  system)  study  of  the  patient  appointment  and  scheduling 
system  at  Keesler.  The  discussion  is  organized  according  to  stages  in 
the  appointing  and  care  delivery  process.  Each  section  describes  the 
activity  involved,  presents  quantitative  findings  and  then  question¬ 
naire  or  interview  responses,  and  finally  discusses  the  conclusions 
derived  from  these  results. 

A.  REQUESTING  AN  APPOINTMENT 


1.  Description  of  the  Process 


During  the  time  of  the  post-implementation  study,  patients  at 
Keesler  Medical  Center  could  make  an  appointment  in  person  and  by 
telephone,  but  most  appointments  were  made  by  telephone.  Patients 
called  either  CAS  or  the  appropriate  clinic,  depending  on  whether 
appointment  scheduling  was  done  centrally  or  in  the  clinic.  Calls 
into  CAS  were  made  either  from  off-base  telephones  or  from  special 
telephones  located  at  convenient  places  around  Keesler.  During  peak 
scheduling  periods  (for  example,  early  morning  hours  or  days  when  new 
schedules  were  made  available),  patients  calling  from  off  base  were 
likely  to  encounter  either  a  busy  signal,  which  indicated  that  all 
telephone  lines  into  CAS  were  busy,  or  a  recorded  message,  which  meant 
that  all  available  telephone  clerks  were  busy.  CAS  was  open  from  0730 
to  1600,  Monday  through  Friday.  Clinics  that  scheduled  their  own 
appointments  did  so  during  normal  clinic  hours  (0800-1600). 

2.  Time  Required  to  Have  a  Call  Answered 

Keesler  Medical  Center  has  an  automated  telephone  monitoring 
system  which  permits  the  supervisor  of  CAS  to  assess,  at  any  time  of 
the  day,  the  number  of  persons  waiting  in  queue  to  have  their  calls 
answered.  The  queue  has  a  limit  of  15  calls  at  any  one  time.  In 
order  to  estimate  the  average  number  of  persons  waiting  in  the  queue, 
data  collectors  queried  the  automated  telephone  system  once  every  half 
hour  from  0730  to  1600  for  a  10-day  period.  Results  are  summarized  in 
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Table  4.  Analysis  is  divided  into  two  types  of  averages:  the  mean 
over  all  observations,  and  average  readings  only  when  a  queue  existed. 

The  percentage  of  observations  where  a  queue  existed  decreased  in 
the  post-implementation  period  from  36%  to  23%.  The  average  number  of 
callers  in  the  queue  over  all  observations  was  1.7  (manual)  and  1.1 
(automated).  The  average  number  of  callers  waiting  in  queue  during 
the  intervals  when  a  queue  existed  was  slightly  less  than  five  in  both 
periods . 

The  telephone  monitoring  system  also  permitted  measurement  of  the 
number  of  seconds  the  first  person  in  the  queue  had  been  waiting. 
This  figure  decreased  from  42  seconds  for  all  baseline  readings  to  18 
seconds  for  post-implementation  readings.  The  average  for  obser¬ 
vations  only  when  a  queue  existed  dropped  from  117  seconds  to  76 
seconds  in  the  post-implementation  study  period. 

3.  Patient  Opinions  About  Telephone  Access  to  Appointments 

a.  Number  of  Attempts  Made  Before  Getting  an  Appointment 

Forty-one  percent  of  the  patients  responding  to  the  Outpatient 
Questionnaire  (see  Appendix  A)  reported  that  they  had  scheduled  their 
appointments  through  CAS;  this  percentage  was  down  from  52%  in  the 
baseline  period.  The  remainder  scheduled  their  appointments  through 
the  clinics.  Table  5  breaks  out  the  calls  made  to  CAS  by  clinic  and 
shows  the  number  of  calls  a  patient  had  to  make  before  getting  through 
to  a  CAS  clerk:  in  the  baseline  period,  44%  of  the  patient  respon¬ 
dents  needed  one  telephone  call  to  make  their  appointments;  16%  called 
twice  and  15%  required  three  calls.  During  the  post-implementation 
evaluation,  the  figures  showed  access  was  not  quite  as  good:  38% 
needed  only  one  call,  19%  required  two  calls,  and  19%  made  three 
attempts  before  getting  through  to  the  appointment  desk.  When  these 
responses  were  broken  down  by  clinic,  the  largest  proportion  in  most 
clinics  made  only  one  attempt  to  make  an  appointment;  in  the  Internal 
Medicine  clinic,  however,  the  largest  percentage  made  more  than  three 
attempts.  These  proportions  were  true  during  the  baseline  evaluation 
as  well. 
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TABLE  4 


AVERAGF  NUMBER  OF  CALLERS  WAITING  AND 
AVERAGE  LENGTH  OF  TIME  TO  REACH  CAS, 

TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


Evaluation 

Base 

Post 

Change 

Percentage  of  time  that  a  queue 
existed,  by  half-hour  segments 
(0730-1600) 

36 

23 

-13  percentage 
points 

Average  number  of  CAS  callers 
waiting  in  queue,  all 
observations 

1.7 

1.1 

-0.6  calls 

Average  waiting  time  in  queue 

for  CAS  caller, 

all  observations  (in  seconds) 

42.1 

17.7 

-24.4  sec 

Average  number  of  CAS  callers 
waiting  in  queue,  when  queue 
existed 

4.8 

4.7 

-0.1  callers 

Average  waiting  time  in  queue 
for  CAS  caller,  when  queue 
existed  (in  seconds) 

116.6 

75.9 

-40.7  sec 

KEY:  Base  =  baseline  (manual  system)  study. 

Post  =  post-implementation  (automated  system)  study. 
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ABLE  5 


NUMBER  OF  ATTEMPTS  BEFORE  GETTING  THROUGH  TO  CAS, 
TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


Genl.  Surg.  Orthopedic 


Number  of 

Keesler 

attempts 

1 

2 

3 

more  than 
3 

can’t 

remember 


Int'l  Med. 
Base  Post 
16.9  21.2 

22.1  15.2 

19.5  21.2 

36.3  40.9 

5.2  1.5 


Prim. 

Care 

Base 

(Post 

1 

52.3 

44.9 

11.9 

18.4 

14.9 

17.2 

17.3 

13.8 

3.6 

5.7 

Base  Post' 
53.8  34.3 

20.5  34.2 

I 

15.4  25.7 

I 

10.3  2.9 

i  l 

I 

0.0  '  2.9 i 


Base  jPost 
51.0  ’56.4 

14.3  15.4 

i 

10.2  12.8 

16.3  12.8 

8.2  2.6 


All 

Base  Post 

44.2  38.4! 

15.6  19.4 

15.3  18.9 

20.7  19.4 

4.2  3.5 


KEY:  Base  *  baseline  (manual  system)  study. 

Post  *  post-implementation  (automated  system)  study. 
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b.  Satisfaction  with  Telephone  System 

Patients  were  asked  to  rate  their  satisfaction  with  the  telephone 
system  for  making  appointments.  In  the  baseline  study,  57%  responded 
that  they  were  satisfied;  61%  expressed  satisfaction  in  the  post- 
implementation  period.  The  telephone  system  was  the  lowest  rated 
aspect  of  the  Keesler  health  care  experience  during  the  baseline 
period;  after  implementation,  only  parking  received  a  lower  rating. 

4.  Provider  Opinions  About  Telephone  System 

Provider  opinions  concerning  the  telephone  system  for  making 
appointments  were  obtained  from  the  Provider  Questionnaire  (see 
Appendix  A).  The  questionnaire  asked  providers  in  Orthopedics, 
General  Surgery,  Internal  Medicine,  Primary  Care,  and  the  Dental 
Clinic  to  rate  their  satisfaction  with  various  aspects  of  the  schedul¬ 
ing  system  and  to  make  suggestions  for  improving  the  system.  Only  27% 
of  the  medical  providers  (in  all  clinics  except  Dental)  were  satisfied 
with  the  telephone  system  during  the  baseline  period,  but  this  per¬ 
centage  rose  to  40%  after  system  implementation.  The  telephone  system 
was  the  least  satisfactory  aspect  rated  by  the  medical  providers  in 
the  baseline  period.  While  the  40%  satisfaction  level  is  still  low, 
it  ranked  higher  than  three  other  facets  (availability  of  x-rays,  same 
day  multiple  clinic  visits,  and  patients  requiring  immediate  appoint¬ 
ments)  of  the  health  care  system  in  the  post-implementation  period. 

5.  Support  Staff  Opinions  About  the  Telephone  System 

Support  staff  opinions  were  obtained  from  the  Appointment  and 
Scheduling  Support  Staff  Questionnaire  (see  Appendix  A). 

Of  the  respondents  in  the  baseline  period,  65%  were  satisfied 
with  the  telephone  system.  This  satisfaction  level  increased  after 
implementation  to  74%.  While  their  satisfaction  with  the  telephone 
system  was  low  relative  to  the  other  aspects  they  rated,  support  staff 
were  less  critical  about  it  than  providers  or  patients. 

6.  Conclusions 

Telephone  access  to  CAS,  as  measured  by  the  telephone  monitoring 
system  (Table  4,  above),  improved  during  the  post-implementation 
period.  Results  from  the  Outpatient  Questionnaire,  however,  show’  that 
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the  percentage  of  callers  that  got  through  to  CAS  on  their  first 
attempt  decreased  by  6%  after  implementation.  In  spite  of  this 
perceived  decrease  in  system  quality,  the  most  consistent  finding 
proved  to  be  the  increase  in  satisfaction  levels  for  patients,  pro¬ 
viders,  and  appointment  staff.  While  the  levels  were  still  not  high 
compared  to  other  aspects  of  the  health  care  system,  all  three  groups 
expressed  greater  satisfaction  with  the  telephone  system  after  imple¬ 
mentation  of  the  automated  system.  Structural  limitations  within  the 
phone  system  may  preclude  very  high  levels  of  satisfaction:  some 
callers  are  always  going  to  be  in  the  queue.  It  is  clear,  however, 
that  since  the  introduction  of  the  TRIPAS  System  telephone  trans¬ 
actions  have  generated  less  dissatisfaction  for  staff,  providers,  and 
patients  alike. 

B.  BEING  SCHEDULED  FOR  AN  APPOINTMENT 


1.  Description  of  the  Process 

A  simple  diagram  of  the  patient  appointment  and  provider  sched¬ 
uling  process  is: 

Care  provider  Medical  Record  delivered 

submits  schedule  to  clinic 


Patient  requests  Medical  Record,  Care  pro- 

an  appointment  vider  and  patient  meet  at 

appointed  time  and  place 

Though  this  process  appears  to  be  straightforward,  in  reality  it 
involves  a  labor-intensive,  time-consuming  set  of  transactions  involv¬ 
ing  multiple  records  of  the  appointments  that  are  made,  cancelled,  and 
changed.  In  addition  to  the  actual  appointment  process  the  appoint¬ 
ment  system  must  interface  with  the  medical  record  system  to  transfer 
a  medical  record  from  the  Medical  Records  Department  to  the  appropri¬ 
ate  clinic  before  the  patient  arrives. 
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next  available  appointment  with  a  given  provider  or  clinic.  The 
software  also  permitted  the  clerk  to  examine  a  given  day's  list  of 
appointments  and  openings.  Once  the  patient's  Social  Security  number 
was  given,  the  clerk  could  search  the  patient  data  base  for  the 
correct  patient  information.  If  a  patient  was  already  registered,  the 
patient  information  needed  to  make  an  appointment  (given  orally  by  the 
caller  under  the  manual  system)  appeared  on  the  screen.  Thus,  the 
potential  existed  for  significant  time  savings  per  transaction  when  a 
patient  was  preregistered.  Illegible  entries  or  misspellings  were 
practically  eliminated  since  no  handwriting  was  involved.  Cancella¬ 
tions  and  changes  in  provider  schedules  were  accomplished  more  easily 
because  they  did  not  require  manual  crossing  out. 

During  the  baseline  evaluation  period  CAS  averaged  roughly  940 
calls  per  day;  during  the  post-implementation  period  the  average  was 
738  (see  Table  6).  This  level  of  activity  is  normally  handled  by  five 
operators.  Each  clerk  is  expected  to  book  about  100  appointments  a 
day  on  average.  During  the  baseline  evaluation  period,  clerks  aver¬ 
aged  200  calls  and  107  appointments  booked  daily.  These  figures 
decreased  in  the  post-implementation  period  to  139  calls  and  73 
appointments  booked  daily.  This  decrease  can  be  explained,  in  part  by 
results  from  our  patient  survey  that  revealed  that  up  to  one-third  of 
patients  appointed  to  clinics  that  booked  through  CAS  were  being 
appointed  instead  by  clinic  personnel  using  the  automated  system. 

b.  Clinics 

The  evaluation  at  Keesler  included  two  specialty  clinics: 
General  Surgery  and  Orthopedics.  These  clinics  booked  their  own 
appointments  and  did  not  utilize  CAS.  For  a  patient  to  be  seen  in 
General  Surgery  and  Orthopedics,  that  individual  first  had  to  be  seen 
in  the  Primary  Care  clinic  and  then  referred  to  the  specialty  clinic. 
Once  they  had  been  referred,  patients  usually  booked  appointments  in 
person  or  by  calling  the  clinic  directly.  The  appointment  clerk 
requested  the  relevant  information  from  the  patient  to  determine  the 
nature  of  the  medical  need.  Care  provider  schedules  were  then  re¬ 
viewed  and  an  appointment  was  scheduled  at  the  earliest  possible  date. 
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At  Keesler  appointments  were  booked  by  either  CAS  or  the  indi¬ 
vidual  clinics.  Different  procedures  were  followed  depending  on 
whether  CAS  or  the  clinic  made  the  appointment. 

a .  CAS 

Under  the  manual  system  at  Keesler,  monthly  provider  schedules 
were  submitted  to  CAS  several  weeks  in  advance.  CAS  staff  maintained 
notebooks  for  each  provider  and  relevant  clinics  that  listed  the 
available  routine  appointment  slots  and  the  time  set  aside  for  acute 
problems.  CAS  also  kept  a  separate  list  of  the  present  and  next  day's 
appointments  and  available  open  slots. 

Under  the  manual  system,  appointment  clerks  at  Keesler  sat  around 
a  rotary  carousel  as  they  received  incoming  calls.  When  an  appoint¬ 
ment  was  requested,  the  operator  searched  for  the  appropriate  notebook 
in  the  carousel,  secured  it  if  it  was  not  being  used  by  another  clerk, 
or  waited  until  it  became  available.  When  a  time  and  day  for  the 
appointment  had  been  settled  on,  the  clerk  asked  a  prescribed  set  of 
questions  (including  registration  information)  necessary  to  complete 
the  transaction.  The  clerks  then  gave  a  set  of  oral  instructions 
about  clinic  locations  and  preparation  the  patient  must  make  and  had 
the  caller  repeat  the  details  of  the  appointment.  The  operator  wrote 
the  necessary  information  into  the  appointment  book  while  speaking 
with  the  caller  and  then  returned  it  to  the  proper  slot  in  the  carou¬ 
sel. 

Two  days  before  the  appointment  date,  the  Daily  Register  was  sent 
by  CAS  to  Medical  Records.  The  Medical  Records  Department  then  pulled 
patient  records  from  the  Daily  Register  patient  appointments  and  sent 
these  registers  and  the  pulled  records  to  the  appropriate  clinic  on 
the  day  before  the  patient's  appointment. 

CAS  clerks  kept  daily  manual  records  of  the  number  and  type  of 
transactions  they  processed.  From  these  records,  supervisors  produced 
summary  documents  of  hospital  appointment  activity  through  CAS. 

Under  the  automated  system,  no  carousel  was  necessary.  The 
operators  had  their  own  stations,  each  equipped  with  a  CRT  terminal. 
When  a  call  was  answered,  operators  used  their  terminal  to  find  the 
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TABLE  6 


NUMBER  OF 

CALLS  RECEIVED  BY 

CAS, 

TRIPAS  EVALUATION, 

USAF  MEDICAL  CENTER 

,  KEESLER  AFB 

Type  of  Call 

Baseline 

Post-impl 

Appointment  Made 

5040 

3861 

Information 

3140 

1601 

Appt .  Unavailable 

823 

1598 

Appt.  Cancelled 

181 

320 

Appt.  Recheduled 

220 

1315 

Total 

9404 

7380 

NOTE:  Because  both  the  baseline  and  post-implementation  data 

collection  period  was  10  working  days,  daily  averages  can  be 
determined  by  dividing  the  total  number  of  monitored  calls  by  10. 
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Priority  was  given  to  active-duty  personnel;  they  were  routinely  seen 
within  24  hours  of  referral. 

Under  the  automated  system,  the  clinics  made  use  of  terminals 
usually  located  at  the  receptionists'  desks.  Receptionists  continued 
to  perform  a  variety  of  tasks,  only  some  of  which  were  related  to 
appointments  and  scheduling.  Information  regarding  appointments  and 
registration  data  were  obtained  by  a  few  keystrokes  in  the  automated 
system.  In  some  cases,  patients  who  were  asked  by  a  physician  to 
return  the  next  day  at  a  particular  time  were  promptly  booked  by  the 
clinic  into  the  automated  system;  under  the  manual  system  this  ar¬ 
rangement  might  have  gone  unnoted.  Other  changes  observed  included 
periodic  use  of  the  clinic  printers  to  produce  lists  of  patients  by 
provider,  and  provider  use  of  the  system  to  check  the  availability 

(for  referrals)  of  open  appointments  for  providers  in  other  clinics. 

2.  Time  Spent  on  Appointment  Activities 

a .  CAS 

Telephone  calls  to  CAS  were  monitored  in  order  to  determine  the 
time  required  to  book  appointments  and  conduct  other  types  of  schedul¬ 
ing  transactions.  Data  summarizing  the  number  of  calls  received  by 
CAS  are  reported  in  Table  6  (above),  and  data  on  the  length  of  moni¬ 
tored  telephone  calls  are  included  in  Table  7. 

The  distribution  of  calls  sampled  by  type  differed  in  the  post¬ 
implementation  period  from  the  baseline  distribution.  There  were 

fewer  appointment-made  and  information  calls  and  twice  as  many  ap¬ 

pointment-unavailable  transactions  in  the  post-implementation  period. 
As  shown  in  Table  6,  the  number  of  information  calls  dropped  sharply. 
This  decrease  may  reflect  the  cyclical  phenomenon  of  the  increased 
experience  patients  had  with  Keesler's  health  care  system  in  May  as 
opposed  to  their  lack  of  familiarity  in  the  preceding  October.  [As  in 
any  military  base,  many  new  families  are  reassigned  to  Keesler  during 
the  summer  months,  and  they  undergo  an  adjustment  period  while  they 
learn  about  how  to  properly  use  base  services.] 

The  average  time  to  complete  a  CAS  telephone  transaction  de¬ 

creased  from  67  to  60  seconds.  All  types  of  CAS  transactions  (except 
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TABLE  7 


LENGTH  OF  TELEPHONE  CALLS  TO  CENTRAL  APPOINTMENT  SECTION, 
TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFE 


No. 

(and  %) 

of  Observations 

Mean 

(seconds ) 

Type  of  Call 

Baseline 

Post 

-impl . 

Baae 

Post 

Appointment  Made 

550 

(65) 

476 

(59) 

82.  1 

78.  1 

Information 

189 

(22) 

129 

(16) 

29.0 

Appt.  Unavailable 

94 

(ID 

191 

(23) 

57.2 

40.0 

Appt.  Cancelled 

7 

(1) 

14 

(1) 

43.0 

36.1 

Appt.  Rescheduled 

6 

(1) 

7 

(1) 

79.5 

94.1 

All  Transactions 

846 

(ion) 

817 

(100) 

67.2 

59.5 

KEY:  Base  =  baseline  (manual  system)  study. 

Post  =  post-implementation  (automated  system)  study. 
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appointment  rescheduling,  which  represented  only  1%  of  all  calls)  took 
less  time  to  complete.  An  appointment  booking  took  4  seconds  less  on 
average  during  the  automated  evaluation  period.  The  greatest  single 
decrease  was  in  calls  where  an  appointment  was  not  available  to  the 
caller,  down  to  40  seconds  on  average  from  57  seconds  in  the  baseline 
period. 

To  make  results  comparable,  the  baseline  distribution  of  tele¬ 
phone  calls  was  used  as  the  standard  to  calculate  and  compare  overall 
averages  of  time  per  CAS  transaction. 

b.  Clinics 

The  average  time  to  complete  a  telephone  transaction  in  the 
Orthopedics  Clinic  (see  Table  8)  decreased  between  the  manual  and 
automated  evaluations  from  147  seconds  to  125  seconds.  Information 
calls  lasted  26  seconds  less  on  average  after  implementation  of  the 
automated  system,  and  booking  an  appointment  took  35  seconds  less. 
The  distribution  of  Orthopedics  phone  calls  by  type  of  call  during  the 
post-implementation  period  was  different  from  the  baseline  period  in 
that  there  were  many  more  appointment-made  transactions  and  fewer 
information  and  appointment-unavailable  calls.  Information  calls 
accounted  for  50%  of  the  transactions  in  the  baseline  period,  and  32% 
in  the  post-implementation  period;  the  percentage  of  appointment-made 
calls  rose  from  32%  to  43%. 

The  General  Surgery  Clinic  (Table  9)  experienced  an  increase  in 
the  time  spent  in  appointment  activities.  The  distribution  of  calls 
was  virtually  the  same  during  both  periods  (information  calls  about 
80%,  appointment  bookings  roughly  17%),  and  the  average  transaction 
time  rose  from  74  seconds  to  86  seconds.  Appointment  bookings  ac¬ 
counted  for  most  of  the  increase.  On  average,  it  took  just  over  ] 
minute  longer  to  book  an  appointment  in  the  General  Surgery  Clinic 
after  implementation  of  the  automated  system  due  to  administrative 
changes  internal  to  this  clinic  that  had  little  to  do  with  TRIPAS. 
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TABLE  8 


LENGTH  OF  TELEPHONE  CALLS  TO  ORTHOPEDICS  CLINIC, 
TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


No. 

(and  %) 

of  Observations 

Mean 

(seconds) 

Type  of  Call 

Baseline 

Post 

-impl. 

Base 

Post 

Appointment  Made 

193 

(32) 

157 

(43) 

201.3 

166.7 

Information 

294 

(50) 

116 

(32) 

113.3 

87.4 

Appt.  Unavailable 

41 

(7) 

65 

(18) 

117.3 

136.8 

Appt.  Cancelled 

29 

(5) 

12 

(3) 

79.0 

135.3 

Appt.  Rescheduled 

36 

(6) 

15 

(4) 

223.4 

193.4 

All  Transactions 

593 

(100) 

365 

(100) 

147.2 

125.0 

KEY:  Base  =  baseline  (manual  system)  study. 

Post  =  post-implementation  (automated  system)  study. 
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TABLE  9 


LENGTH  OF  TELEPHONE  CALLS  TO  GENERAL  SURGERY  CLINIC, 
TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


No. 

(and  %) 

of  Observations 

Mean 

(seconds] 

Type  of  Call 

Baseline 

Post 

-impl . 

Base 

Post 

Appointment  Made 

76 

(17) 

74 

(16) 

130.4 

192.2 

Information 

359 

(80) 

370 

(81) 

61.4 

63.8 

Appt .  Unavailable 

7 

(2) 

4 

(1) 

115.  1 

70.3 

Appt .  Cancelled 

6 

(1) 

5 

(1) 

46.8 

67.8 

Appt.  Rescheduled 

1 

(1) 

2 

(1) 

180.0 

134.0 

All  Transactions 

449 

(100) 

455 

(100) 

74.0 

86. 1 

KEY:  Base  =  baseline  (manual  system)  study. 

Post  =  post-implementation  (automated  system)  study. 
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3.  Staff  Productivity 

The  number  of  telephone  transactions  in  the  CAS  was  recorded  by 
type  of  call  on  workload  logs  filled  out  by  each  clerk  in  the  baseline 
period.  A  combination  of  logs  and  TRIPAS  System  reports  was  used  to 
compile  post-implementation  figures. 

The  average  number  of  CAS  telephone  transactions  per  day  on  the 
manual  system  was  940  (see  Table  6,  above).  This  figure  declined 
after  implementation  of  the  automated  system  to  738  calls  per  day. 
The  average  number  of  appointment  bookings  per  day  dropped  from  504 
(baseline)  to  386  (post-implementation).  The  three  highest  baseline 
figures  for  appointments  booked  per  day  by  a  single  CAS  clerk  were 
251,  195,  and  170.  The  three  highest  corresponding  post-imple¬ 
mentation  figures  were  149,  145,  and  144.  The  mean  number  of  trans¬ 
actions  per  clerk  per  hour  decreased  from  27  to  19  between  evaluation 
periods.  These  data  reflect  the  smaller  volume  of  incoming  calls, 
rather  than  a  decline  in  productivity. 

4.  Patient  and  Staff  Opinions 

a.  Patients 

Outpatients  in  the  four  sample  clinics  were  asked  their  opinions 
of  the  appointment  and  scheduling  system.  They  were  asked  basic 
information  about  how  and  when  their  appointments  were  made,  as  well 
as  their  degree  of  satisfaction  with  the  various  aspects  of  the  health 
care  system.  (See  Appendix  A  for  a  composite  of  the  results  of  this 
questionnaire  and  Table  10  for  selected  results.) 

When  asked  when  they  had  scheduled  their  appointments,  56%  of  the 
patients  in  the  baseline  period  replied  that  they  had  booked  them  a 
week  or  less  in  advance,  compared  to  47%  of  the  post-implementation 
respondents.  The  percentage  of  patients  who  had  made  their  appoint¬ 
ments  more  than  2  weeks  before  was  28%  (baseline)  and  27%  (post¬ 
implementation).  About  70%  of  respondents  from  both  study  periods 
would  have  preferred  an  appointment  sooner,  and  there  was  an  approxi¬ 
mately  2.92%  increase  (from  6.3%  to  9.2%  observed  for  the  post¬ 
implementation  period)  in  the  number  of  patients  who  had  same  day 
multiple  clinic  appointments. 
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Patient  satisfaction  with  the  appointment  and  scheduling  system 
in  general  rose  in  each  of  the  four  clinics.  The  satisfaction  level 
for  the  clinics  taken  as  a  whole  increased  by  7%  (Table  10).  Patients 
were  asked  to  rate  their  satisfaction  with  three  specific  aspects  that 
directly  contributed  to  their  satisfaction  with  the  appointment  system 
in  general:  ease  of  booking  appointments,  availability  of  appointment 
at  time  desired,  and  performance  of  the  telephone  system.  While 
satisfaction  with  ease  of  booking  and  with  availability  of  appoint¬ 
ments  at  the  time  desired  remained  the  same,  telephone  satisfaction 
improved  and  may  have  contributed  to  the  7%  increase  in  satisfaction 
with  the  system  as  a  whole.  The  clinic  that  showed  the  most  improve¬ 
ment  (Internal  Medicine)  had  the  lowest  baseline  satisfaction  ratings. 
That  clinic's  level  of  satisfaction  with  the  telephone  system  and  the 
appointment  and  scheduling  process  increased  from  47%  to  55%  and  from 
54%  to  63%,  respectively. 

The  Dental  Clinic  was  evaluated  for  patient  satisfaction.  A 
discussion  of  these  data  is  included  in  Appendix  B. 
b.  Providers 

Opinions  about  baseline  scheduling  services  were  obtained  from  52 
medical  providers  in  four  clinics  using  the  Provider  Questionnaire 
(see  Appendix  A).  The  same  survey  questions  were  also  asked  of  35 
providers  in  the  post-implementation  study.  In  the  baseline  study, 
providers  were  asked  to  set  goals  for  provider  satisfaction  with  the 
overall  quality  of  the  appointment  system.  They  set  a  minimum  level 
of  57%  satisfied  with  the  system,  a  standard  of  75%  and  a  maximum 
expected  level  of  87%.  Table  li  shows  provider  opinion  about  appoint¬ 
ment  services  in  the  two  study  periods. 

The  most  striking  improvement  (21%)  was  in  the  providers'  per¬ 
ceptions  that  patients  were  having  an  easier  time  booking  appointments 
with  the  automated  system.  Providers  also  felt  that  more  patients 
could  book  appointments  when  they  wanted  (15%),  and  that  the  telephone 
system  had  improved  (13%).  Although  providers  also  felt  that  patients 
were  less  able  to  book  an  appointment  on  an  immediate  basis  (a  de¬ 
crease  of  11%),  overall  their  levels  of  satisfaction  with  the 
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PATIENT  SATISFACTION  WITH  APPOINTMENT  SYSTEM, 
TRIPAS  EVALUATION,  IISAF  MEDICAL  CENTER,  KEESLER  AFB 
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KEY:  Base  =  baseline  (manual  system)  study. 

Post  =  post-implementation  (automated  system)  study. 


TABLE  11 


PROVIDER  SATISFACTION  WITH  APPOINTMENT  SYSTEM, 
TRIP AS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


Impact  Area 


Patients'  ease  of  booking  appts. 


Patients'  ability  to  book  appts. 
at  time  desired 


Waiting  time  after  arrival 

Telephone  system  for  making  appts. 

Accuracy  of  assigning  patients  to 
appropriate  clinic 

Ability  of  appointment  and 

scheduling  system  to  accommodate: 

Flexible  schedules 
Patients  requiring  immediate 
appointments 

Importance  of  appointment  and 
scheduling  services 

Overall  quality  of  appointment  and 
scheduling  services 


Net  Percent  Satisfied 


Baseline 

Post 

Change 

39 

60 

+21 

percentage 

points 

40 

55 

+  15 

percentage 

points 

75 

80 

+5 

percentage 

points 

27 

40 

+  13 

percentage 

points 

70 

76 

+6 

percentage 

points 

45 

53 

+7 

percentage 

points 

42 

31 

-11 

percentage 

points 

92 

96 

+4 

percentage 

points 

49 

63 

+  14 

percentage 

poirts 

NOTE:  The  number  of  respondents  in  the  baseline  period  was  52;  in  the 
post-implementation  period,  35. 


/L  Arthur  D.  Little,  Inc 


33 


-.'V- 


appointment  and  scheduling  system  increased  comfortably  with  the 
automated  system,  from  49%  satisfied  in  the  baseline  period  to  63% 
with  the  TRIPAS  System. 

Providers  were  queried  as  to  how  important  they  felt  the  patient 
appointment  and  scheduling  system  was  to  the  quality  of  care  delivered 
at  Keesler.  Of  the  26  providers  who  responded  to  this  question,  96% 
felt  it  was  important;  only  one  disagreed. 

Provider  comments  about  the  TRIPAS  System,  supplied  in  the 
"Additional  comments"  section  of  the  survey  and  in  personal  inter¬ 
views,  were  also  supportive:  "It  increases  my  efficiency.  Since  I 
don't  have  a  secretary,  it  makes  a  big  difference  to  me.  I'm  happy 
with  the  thing."  Another  provider  commented  that  TRIPAS  had  "made  it 
easier  for  me  to  anticipate  the  next  day's  workload." 

Opinions  of  providers  in  the  Dental  Clinic  were  also  solicited; 
responses  from  that  clinic  are  in  Appendix  B. 

c.  Support  Staff 

Support  staff  were  surveyed  about  the  importance  of  the  appoint¬ 
ment  and  scheduling  system  to  the  quality  of  care,  their  satisfaction 
with  the  various  aspects  of  the  appointment  and  scheduling  system,  and 
their  suggestions  for  improving  the  system.  Table  12  shows  support 
staff  opinions  about  the  appointment  system  in  the  two  study  periods. 
Every  aspect  examined  showed  an  improvement  over  the  baseline  period. 
The  most  notable  changes  were  the  ability  of  the  system  to  change 
provider  schedules  and  to  enable  patients  to  book  appointments  when 
they  wanted,  but  the  overall  consistent  and  steady  improvement  in  all 
areas  is  perhaps  the  most  striking  aspect. 

Some  staff  comments  on  the  automated  system  are  revealing:  the 
chief  nurse  in  Primary  Care  said,  "It's  improved  the  utilization  of 
appointment  times,  and  it's  improved  accountability.  But  we  have 
ended  up  making  more  appointments  than  we  should  be  making."  The 
manager  of  CAS  likewise  commented,  "The  PAS  System  has  greatly  im¬ 
proved  the  efficiency  of  obtaining  available  appointments,  and  has 
been  advancageous  to  patients  and  providers.  Appointments  can  be  made 
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TABLE  12 


SUPPORT  STAFF  SATISFACTION  WITH  APPOINTMENT  SYSTEM, 
TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


Net  Percent 

Satisfied 

Impact  Area 

Baseline 

Post 

Change 

Patients'  ease  of  booking  appts. 

63 

82 

+19  percentage  points 

Patients'  ability  to  book  appts. 

at  time  desired 

55 

85 

+30  percentage  points 

Telephone  system  for  making  appts. 

65 

74 

+9  percentage  points 

Ability  of  appointment  and 

scheduling  system  to  accommodate 

: 

Flexible  schedules 

44 

76 

+32  percentage  points 

Patients  requiring  immediate 

appointments 

66 

82 

+16  percentage  points 

Relative  ease  for  staff  to: 

Book  appointments 

77 

91 

+14  percentage  points 

Cancel  appointments 

74 

97 

+23  percentage  points 

Reschedule  appointments 

77 

88 

+11  percentage  points 

Dispense  information  to  patients 

69 

76 

+7  percentage  points 

Importance  (%  of  respondents 

feeling  system  is  important) 

93 

94 

+1  percentage  points 

Overall  satisfaction 

75 

88 

+13  percentage  points 

NOTE:  The  number  of  respondents  in  the  baseline  period  was  41;  in  the 
post-implementation  period,  34. 
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in  more  than  one  location.  This  allows  greater  access  for  everybody 
involved . " 

5.  Conclusions 

The  TRIPAS  System  was  expected  to  decrease  the  time  spent  in 
appointment  activities.  The  data  collected  in  the  Central  Appointment 
Section  support  this  hypothesis.  The  overall  transaction  time  within 
CAS  decreased  an  average  of  7  seconds  per  call.  Data  from  the  clinics 
gave  mixed  results:  in  one  clinic  transaction  time  increased,  in 
another  it  decreased.  Patients,  providers,  and  support  staff  reported 
greatly  increased  satisfaction  with  the  appointment  process. 

During  the  post-implementation  study  period,  CAS  received  fewer 
calls  than  in  the  baseline  study  period.  Because  of  this,  impact  of 
the  reduced  transaction  time  on  improved  peak  period  performance  or 
better  phone  access  to  CAS  could  not  be  documented.  But  it  can  be 
assumed  that  during  a  period  of  increased  workload  the  shorter  trans¬ 
action  times  with  TRIPAS  would  result  in  more  efficient  and  expedi¬ 
tious  clerk  performance. 

The  opinions  of  the  support  staff  provide  some  insight  into  this 
same  issue  (decrease  in  telephone  calls  to  CAS).  It  seems  likely  that 
more  appointments — particularly  follow-up  visits — are  being  scheduled 
by  staff  at  individual  clinics,  even  when  those  clinics  normally  use 
CAS;  the  increased  ease  of  booking  appointments  under  TRIPAS  may  be  a 
factor  in  this  shift. 

C.  HAVING  THE  MEDICAL  RECORD  LOCATED  AND  SENT  TO  THE  APPROPRIATE 

CLINIC 

1.  Description  of  the  Process 

Under  the  manual  system  the  Medical  Records  Department  at  Keesler 
had  three  shifts:  day  (0800-1600),  night  (1600-2400),  and  midnight 
(2400-0800).  The  morning  shift  pulled  the  medical  records  for  after¬ 
noon  appointments  on  the  same  day,  and  the  midnight  shift  pulled  the 
medical  records  for  appointments  the  next  morning.  Clinic  staff 
picked  up  the  records  for  their  respective  clinics.  CAS  clerks 
completed  the  Medical  Record  Request  Form  (Form  250)  for  all  persons 
to  be  seen  in  Primary  Care.  These  forms  were  left  in  a  box  in  CAS  to 
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be  picked  up  daily  by  a  staff  person  from  the  Medical  Records  Depart¬ 
ment.  All  Form  250s  for  afternoon  appointments  were  picked  up  by 
10:00  a.m.  so  that  records  could  be  forwarded  to  Primary  Care. 

The  installation  of  TRIPAS  resulted  in  a  reorganization  of  duties 
among  the  three  shifts;  with  TRIPAS,  the  day  shift  pulled  medical 
records  for  walk-in  patients;  the  night  shift  put  away  records  from 
the  day's  appointments;  and  the  midnight  shift  pulled  records  for  the 
next  day's  appointments.  With  the  TRIPAS  System  the  Medical  Records 
Department  generated  the  Medical  Records  pull  list  for  each  clinic  16 
hours  in  advance  of  the  appointment  day.  The  pull  list  consists  of 
three  components:  a  written-out  list  by  provider  and  clinic;  charge- 
out  cards;  and  provider  schedule  by  provider  and  clinic. 

2.  Availability  of  Medical  Records 

Data  on  the  availability  of  medical  records  at  the  time  of  the 
patient's  appointment,  which  were  collected  during  both  study  periods, 
are  shown  in  Table  13.  Overall,  medical  records  were  available  for 
87%  of  the  patients  observed  during  the  post-implementation  period  as 
opposed  to  85%  during  the  baseline  period.  Internal  Medicine  had  the 
highest  medical  record  availability  (96%)  during  the  post-implementa¬ 
tion  period.  This  represents  a  significant  increase  over  the  baseline 
period,  where  Internal  Medicine  had  a  medical  record  availability  of 
89%.  General  Surgery  also  had  an  increase  in  medical  record  availa¬ 
bility  during  the  post-implementation  period.  Primary  Care  record 
availability  rose  slightly  during  the  same  period.  Orthopedics 
experienced  a  drop  in  medical  record  availability  during  the  post¬ 
implementation  period. 

3.  Patient  and  Staff  Opinions 

a.  Patients 

Patients  were  asked  whether  their  medical  records  were  available 
for  their  clinic  visit  and  where  the  records  were  located  (at  the 
clinic  when  they  arrived,  picked  up  at  Medical  Records,  or  brought 
from  home).  Patients  were  also  asked  about  their  satisfaction  with 
the  availability  of  medical  records  and  were  asked  to  comment  about 
record  availability.  Results  were  obtained  from  the  post- 
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TABLE  13 


MEDICAL  RECORD  AVAILABILITY  FOR  APPOINTED  PATIENTS  IN  STUDY  CLINICS, 
TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


Clinic 


Booked  by  Clinic: 

Orthopedics 
General  Surgery 

Booked  by  CAS : 

Internal  Medicine 
Primary  Care 


Total  for  all 
respondents 


KEY:  Base  =  baseline  (manual  system)  study. 

Post  =  post-implementation  (automated  system)  study. 
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implementation  Outpatient  Questionnaire,  which  showed  an  87%  medical 
record  availability.  The  table  below  summarizes  the  results  from  the 
Outpatient  Questionnaire  as  to  medical  record  and  x-ray  availability 
for  both  periods: 

Post- 

Baseline  impl .  Change 

Medical  Record  Available  85%  87%  +2  percentage  points 

X-Ray  Available  63%  68%  +5  percentage  points 

b.  Providers 

Medical  care  providers  were  asked  to  rate  their  satisfaction  with 
the  availability  of  medical  records  at  the  time  of  their  patients' 
appointments.  Their  satisfaction  rating  showed  an  increase  from  69% 
in  the  baseline  period  to  72%  in  the  post-implementation  period. 

c.  Support  Staff 

Interviews  with  Medical  Records  staff,  although  they  did  not 
provide  quantifiable  data,  indicated  that  the  automated  pull  list, 
whose  hierarchy  of  numbers  corresponded  with  the  sequence  of  record 
filming,  was  far  superior  to  the  manual  chits  and  that  the  automated 
charge-out  cards  improved  efficiency. 

4.  Conclusions 

The  two  measures  of  medical  record  availability,  the  Waiting  Time 
Record  and  the  Outpatient  Questionnaire,  were  remarkably  consistent  in 
both  the  baseline  and  post-implementation  periods.  Both  showed  an  85% 
availability  rate  in  the  baseline  period,  and  both  were  about  87% 
during  the  automated  system  evaluation.  This  represents  little  change 
for  the  four  clinics  taken  as  a  whole. 

There  were  definite  changes,  however,  in  the  individual  rates. 
The  Orthopedics  Clinic  experienced  a  decrease  in  availability;  the 
rate  in  Primary  Care  stayed  roughly  the  same,  while  Surgery  and 
Internal  Medicine  experienced  increases.  The  post-implementation 
Internal  Medicine  availability  rate  is  the  only  one  that  exceeds  or 
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meets  the  standard  level  of  performance  (91%)  establish'd  by  provider 
estimates . 


D,  RECEIVING  CARE  AT  THE  APPOINTED  TIME 

1.  Description  of  the  Process 

At  Keesler,  almost  all  outpatient  appointments  were  scheduled  for 
a  specific  time  slot.  Most  patients  arrived  in  the  clinic  several 
minutes  before  their  appointment  time.  After  presenting  themselves  to 
the  receptionist,  they  waited  to  be  called  by  the  care  provider  or 
received  some  preliminary  tests  or  procedures  from  allied  health  care 
personnel.  After  receiving  these  tests,  they  again  waited  for  the 
health  care  provider.  Patients  without  scheduled  appointments  (walk- 
ins)  were  processed  in  a  similar  fashion. 

2.  Clinic  Workload 

The  number  of  cancellations  and  no-show,  walk-in,  and  appointed 
patients  were  monitored  for  10  days  in  the  four  sample  clinics. 
Seventy-seven  percent  of  patients  seen  during  the  baseline  period  had 
scheduled  appointments;  this  figure  dropped  after  implementation  to 
69%.  In  the  baseline  period,  6.0%  did  not  show  up  and  did  not  cancel 
their  appointments.  This  no-show  rate  decreased  very  slightly  after 
automation  to  5.8%  —  not  a  statistically  significant  decrease. 


show 

rate : 

Baseline 

Impl . 

• 

Primary  Care 

5% 

4% 

• 

General  Survey 

3% 

11% 

• 

Orthopedics 

12% 

8% 

• 

Internal  Medicine 

6% 

8% 

• 

KMC  4  clinics  average 

6% 

5.8% 

3.  Patient  Waiting  Time 

Patient  waiting  times  were  measured  for  a  total  of  464  patients. 
Seventy-three  percent  of  these  patients  had  appointments  while  27% 
were  walk-ins.  The  table  below  illustrates  the  breakdown  of  appointed 
patients  versus  walk-in  patients  by  clinic  after  implementation  of  the 
automated  system: 
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Clinic 


%  Appointed 


%  Walk-ins 


Primary  Care 

89 

11 

General  Surgery 

86 

14 

Orthopedics 

70 

30 

Internal  Medicine 

46 

54 

Table  14  shows  the  frequency  distribution  by  clinic  for  both 
study  periods.  Table  15  compares  the  average  waiting  times  by  clinic 
for  both  periods.  As  in  the  baseline  period,  waiting  time  was  defined 
as  follows:  for  appointed  patients,  the  time  of  appointment  to  the 
time  the  patient  was  seen  by  the  physician  in  his  office;  for  walk-in 
patients,  the  time  the  patient  arrived  in  the  clinic  to  the  time  he 
was  seen  by  the  physician.  For  scheduled  patients  who  were  seen  by 
the  physician  before  their  actual  time  of  appointment,  waiting  time 
was  counted  as  zero. 

For  all  patients,  the  average  waiting  time  in  the  four  clinics 
decreased  by  an  average  of  3  minutes.  For  active-duty  patients,  the 
average  waiting  time  decreased  by  an  average  of  4  minutes  (from  28 
minutes  down  to  24  minutes),  with  the  biggest  improvement  occurring  in 
Internal  Medicine.  Non-active-duty  patient  waiting  time  decreased  by 
1.24  minutes,  from  30.75  minutes  to  29.51  minutes. 

The  percentage  of  patients  seen  within  30  minutes  of  their 
arrival  in  the  clinic  also  improved  significantly  after  implementation 
for  all  clinics: 


Clinic 

Baseline 

Post-impl. 

Change 

Orthopedics 

39% 

48% 

+9% 

General  Surgery 

71% 

80% 

+9% 

Internal  Medicine 

60% 

89% 

+29% 

Primary  Care 

72% 

79% 

+7% 

Twelve  patients  (3% 

of  the  total  sample  and  4%  of  all 

appointed 

patients)  were  late  for 

their  appointments 

and  were  excluded 

from  the 

analysis  performed  in  Table  14. 
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PATTENT  WAITING  TIMES  IN  STUDY  CLINICS, 

TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 
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TABLE  15 


AVERAGE  PATIENT  WAITING  TIKES  BY  CLINIC, 

TRIPAS  EVALUATION,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


Clinic 

Waiting 

Baseline 

Time  (in  minutes) 
Post-impl.  Change 

Orthopedics 

50 

47 

-3 

General  Surgery 

23 

21 

_2 

Internal  Medicine 

22 

11 

-11 

Primary  Care 

23 

23 

0 

Overall  Average 

29.3 

26.4 

-2.9 
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In  the  post-implementation  study,  the  number  of  walk-in  patients 
seen  in  all  four  clinics  totalled  124.  Their  average  waiting  time 
varied  from  a  low  of  18.6  minutes  in  General  Surgery  to  a  high  of  85.7 
minutes  in  Orthopedics.  In  general,  the  average  waiting  time  for 
walk-in  patients  improved  in  the  clinics  except  for  Orthopedics.  The 
average  waiting  times  for  walk-in  patients  in  each  of  the  four  clinics 


in  each  period  are  as  follows 

: 

V 

Clinic 

Baseline 

Post-impl. 

Change 

Orthopedics 

64  min 

86  min 

+22  min 

v 

General  Surgery 

28  min 

19  min 

-9  min 

o 

Internal  Medicine 

63  min 

55  min 

-8  min 

Primary  Care 

N.  A. 

38  min 

N.  A. 

a 

The  percentage  of  walk- 

-in  patients  seen 

within  30 

minutes  of 

their  arrival  in  the  clinic 

likewise  improved 

with  the  exception  of 

Orthopedics: 

E 

Clinic 

Baseline 

Post-impl . 

Change 

k** 

Orthopedics 

25% 

8% 

-17% 

General  Surgery 

72% 

93% 

+21% 

Internal  Medicine 

36% 

39% 

+3% 

Primary  Care 

N.  A. 

50% 

N.A. 

Data  were  collected  on  the  number  of 

incorrectly 

appointed 

patients  at  each  of  the  4  sample  clinics.  These  were  patients  who 
said  they  had  appointments,  but  were  not  on  the  daily  appointment 
lists.  These  patients  were  usually  treated  as  if  they  were  walk-ins. 
The  number  of  errors  decreased  from  201  in  the  baseline  to  34  during 
the  post-implementation  period. 

4.  Patient  Opinions 

Patients  were  asked  to  rate  their  satisfaction  with  clinic 
waiting  time  on  the  Outpatient  Questionnaire.  The  percentage  of 
respondents  who  were  very  or  somewhat  satisfied  was  nearly  the  same 
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both  before  and  after  implementation.  Sixty-three  percent  were 
satisfied  in  the  baseline  period  compared  to  64%  after  implementation. 
E.  SYSTEM  COSTS 

1.  Costs 

Three  sizes  of  the  TRIPAS  System  will  support  the  large,  medium, 
and  small  MTF.  They  differ  in  the  capacity  of  the  main  memory  of  the 
CPU  and  its  associated  hardware  to  support  MTF  workload.  Of  the 
proposed  initial  procurement  of  12  systems,  four  of  the  TRIPAS  sites 
are  considered  large  according  to  the  definition,  five  are  medium,  and 
three  are  small.  For  purposes  of  distributing  front-end  nonrecurring 
costs  and  simplifying  the  analysis,  it  is  assumed  that  the  option  to 
purchase  all  12  TRIPAS  Systems  will  be  exercised  concurrently  and  that 
the  systems  have  an  operating  life  of  8  years. 

Table  16  shows  nonrecurring  costs  for  each  site  regardless  of 
size.  These  costs  reflect  vendor  prices  as  specified  in  the  bid  of 
the  prime  contractor,  but  they  have  been  increased  to  cover  costs 
associated  with  a  contract  modification  to  that  original  bid.  As 
indicated,  the  nonrecurring  costs  (per  site  regardless  of  size)  total 
$171,550. 

Annual  recurring  costs  and  nonrecurring  costs  for  communications, 
lines,  and  modems  differ  for  the  three  MTF  sizes,  as  shown  in 
Table  17.  Assuming  nonrecurring  costs  are  annualized,  the  cost  per 
year  for  TRIPAS  ranges  from  $106,612  for  a  small  site  to  $203,955  for 
a  large  MTF  such  as  Keesler. 

2.  Cost  per  Workload  Unit 

Two  measures  of  workload  were  considered  appropriate  units  to 
analyze  the  impact  of  TRIPAS  on  costs: 

1.  Annual  outpatient  encounters; 

2.  Total  annual  appointment  and  administrative  transactions 
associated  with  these  outpatient  encounters. 

Outpatient  encounters  are  scheduled  visits,  walk-ins,  or  telephone 
consultations.  In  both  the  manual  and  automated  systems,  names  of 
walk-ins  and  telephone  consultation  patients  are  added  to  daily  clinic 
lists  of  appointed  patients  who  keep  their  appointments.  At  the  end 
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TABLE  16 


NONRECURRING  COSTS  OF  TRI-SERVICE  PATIENT 
APPOINTMENT  AND  SCHEDULING  SYSTEM 


Cost  Element 

Cost 

Software  fee 

$  7,500 

Software  license 

76,250 

Training 

14,800 

Documentation 

3,000 

Transport at ion/ Installation 

25,000 

Systems  Implementation 

15,000 

Medical  Treatment  Recording  Card 

25,000 

Maintenance  Outside  Principal  Period 
of  Maintenance 

5,000 

Total 

$171,500 

NOTE:  These  non-recurring  costs  are  not  size  dependent. 


/ti  Arthur  D.  Little,  Inc. 


46 


TABLE  17 


COSTS  OF  TRI-SERVICE  PATIENT  APPOINTMENT  AND  SCHEDULING  SYSTEM 
FOR  LARGE,  MEDIUM,  AND  SMALL  MEDICAL  TREATMENT  FACILITIES 


Cost  Element 
NONRECURRING® 

Non-size-dependent  costs  $  21,444  $  21,444^  $  21,444 

Communications,  lines,  modems  20,625  14,875°  9,125 

Total  nonrecurring  costs  $42,069  $36,319  $30,569 

RECURRING 

Hardware: 

CPU 
CRT 

On-line  storage 
Off-line  storage 
Line  printer 
Character  printer 
Communications  interface 
Operator's  console 

Maintenance: 


Software 

26,800 

26,800 

26,800 

Hardware 

55,742 

29,810 

19,026 

Communications,  lines,  modems 

14,508 

10,463 

Medical  treatment  card 

3,000 

3,000 

3,000 

Total  recurring  costs 

$161,886 

$111,605 

$  76,043 

TOTAL  COST 

$203,955 

$147,924 

$106,612 

Annualized  over  8-year  life  cycle. 

^Estimated  from  large  and  small  systems  costs, 
c 

Cost  included  in  hardware  maintenance. 


$  26,730 
18,492 
3,900 

1,290 

5,520 

5,586 

318 


$  10,842 
14,628 
6,984 
1,620 
1,290 
1,932 
3,918 
318 


$  7,338 

6,649 
5,484 
1,620 
1,290 
1,932 
2,586 
318 


Annual  Cost,  by  Size  of  MTF 
Large  Medium  Small 
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of  the  day,  these  lists  are  used  to  count  clinic  workload.  TRIPAS 
automatically  counts  workload  and  produces  statistical  reports  for 
administrative  review. 

For  every  outpatient  visit,  there  is  more  than  one  appointment/ 
administrative  transaction.  These  transactions  include  registration, 
appointment  bookings,  cancellations,  and  several  other  types  of 
information  inquiries:  confirmation/verification,  appointments  that 
cannot  be  booked  for  a  variety  of  reasons,  and  responses  to  other 
questions  about  provider  schedules  and  availability  of  appointments. 

An  accurate  estimate  of  outpatient  encounters  was  obtained  from 
Keesler  and  an  approximation  of  the  number  of  transactions  discussed 
above  was  calculated.  According  to  Keesler  workload  reports,  the 
number  of  outpatient  encounters,  including  dental,  in  FY  1982  was 
438,600.  The  difficulty  in  determining  the  number  of  transactions 
lies  in  the  fact  that  data  on  transactions  are  available  only  for  CAS, 
which  does  not  account  for  all  appointment  transactions.  The  proce¬ 
dure  described  below  was  used  to  develop  an  overall  estimate  of  trans¬ 
actions  on  the  basis  of  total  outpatient  encounters  and  CAS  workload 
data. 

According  to  pre-  and  post-implementation  data,  CAS  at  Keesler 
experienced  an  average  annual  workload  of  218,979  calls,  classified  as 
follows : 

Appointment  booked  through  CAS  (including  119,016 

registration  and  registration  verification) 

Cancellations  through  CAS  6,525 

Information  from  CAS  (including  appointment  93,438 

not  available) 

This  information  yields  a  ratio  of  1.84  transactions  per  CAS  booked 
appointment  (119,016  +  6,525  +  93,438/119,016). 

The  total  number  of  visits  made  by  appointment,  including  those 
booked  by  the  clinics,  as  well  as  by  CAS,  was  determined  by  taking 
total  outpatient  visits  and  eliminating  clinic  walk-ins,  visits  to  the 
emergency  room  and  dispensary,  and  sick-call  clinics.  Estimates  of 
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these  were  obtained  from  workload  data  for  a  previous  evaluation  which 
determined  that  39.2%  of  outpatient  visits  were  for  these  types  of 
visits.  Using  the  converse  of  these  percentages,  total  appointed 
visits  came  to  266,669  (438,600  x  60.8%).  Using  the  1.84  ratio,  these 
visits  accounted  for  490,671  transactions.  The  number  of  clinic  walk- 
ins  and  visits  to  ER/AMIC  (Emergency  Room/Acute  Minor  Injury  Clinic) 
is  438,600  less  266,669,  or  171,931.  Each  of  these  visits  represents 
a  transaction,  because  each  requires  registration  verification  and 
logging  for  workload  reporting.  Thus,  the  total  annual  number  of 
transactions  is  estimated  to  be  490,671  plus  171,931,  or  662,602. 

Dividing  these  annual  workload  units  into  the  annual  costs  of 
using  a  large-sized  TRIPAS  system  ($203,955)  yields  a  cost  per  out¬ 
patient  visit  of  46.5  cents  and  cost  per  transaction  of  30.8  cents. 
For  manual  operation  of  the  appointment  and  scheduling  system,  the 
cost  per  CAS  transaction  was  calculated  to  be  43  cents.  Therefore, 
before  deducting  any  savings  attributable  to  TRIPAS,  the  raw  cost  of 
operating  the  TRIPAS  System  approximately  doubles  the  cost  of  appoint¬ 
ment  operations,  to  74  cents  per  transaction. 

Further  analysis  of  costs  and  potential  offsetting  cost  savings 
are  assessed  in  a  separate  document  on  economic  analysis. 
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IV.  GOALS  AND  OBJECTIVES 


In  order  to  evaluate  the  TRIP AS  System,  the  performance  of  TRIPAS 
at  Keesler  was  compared  to  the  system  goals  and  objectives.  A  summary 
of  this  comparison  is  contained  in  Exhibit  A.  Three  factors  should  be 
considered  in  assessing  TRIPAS  performance  at  Keesler.  First,  the 
data  available  in  the  evaluation  are  subject  to  two  constraints.  One 
is  that  baseline  and  post-implementation  data  were  collected  at 
different  times  of  the  year;  the  baseline  data  were  collected  in 
October  and  the  post-implementation  data  in  May.  In  general,  outpa¬ 
tient  appointment  activities  declined  during  the  summer  months  and,  by 
the  end  of  May,  it  is  likely  that  that  dropoff  would  already  have 
started.  In  fact,  there  was  a  lower  level  of  appointing  activity 
occurring  at  the  time  of  the  post-implementation  study. 

The  second  constraint  was  that  the  data  in  the  post-implemen¬ 
tation  study  were  collected  late  in  the  month,  whereas  the  baseline 
data  were  collected  throughout  the  month  of  October.  Appointment 
activities  tend  to  cluster  at  the  beginning  of  the  month  because  most 
clinics  make  their  schedules  available  at  that  time.  These  two 
factors  probably  contributed  to  a  decline  in  appointing  activities  and 
a  decline  in  patients  presenting  for  appointments  in  the  post¬ 
implementation  study. 

Also,  the  TRIPAS  System  had  been  operational  for  only  3  months  at 
the  time  of  the  post-implementation  data  collection;  therefore, 
although  its  immediate  benefits  could  be  assessed,  some  long-term 
benefits  that  depend  on  changing  attitudes  or  management  practices 
will  probably  appear  only  later  in  the  system's  lifecycle.  There  are 
also  some  benefits  that  accrue  in  accordance  with  user  experience  with 
the  system.  These  also  may  not  have  been  fully  realized.  In  some 
ways,  Keesler's  appointment  system  may  be  unique  in  that,  in  areas 
where  the  performance  was  already  high,  TRIPAS  may  not  have  had  as 
large  an  impact  at  Keesler  as  it  would  at  a  more  typical  facility. 
All  these  factors  will  be  discussed  relative  to  each  goal  and  objec¬ 
tive  . 
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EXHIBIT  A 


COMPARISON  OF  TRIPAS  PERFORMANCE  TO  SYSTEM  COALS  AND  OBJECTIVES 


Goal 

Status  at 
Implementation 

Projected  Change 

Reduced  time  to  make  a 
patient  appointment 

Realized  in  CAS 

Realize  throughout 

Reduced  clinic  waiting 
time 

Realized 

Improve  further  with 
improved  provider  access 

Reduced  trips  to  receive 
care  in  multiple  clinics 
on  same  day 

Realized 

Improve  with  improved 
provider  access,  knowl¬ 
edge  of  system  capa¬ 
bilities 

Reduced  appointment 
clerical  work 

Realized 

No  change 

Increased  utilization  by 
better  matching  patients, 
providers,  facilities 

Realized 

Improve  further  with 
management  intervention 

Reduced  "no-shows" 

Not  realized 

Improve  with  management 
intervention 

Reduced  appointment 
transcription  and 
processing  errors 

Realized 

Improve  further  with 
database  maintenance 

Improved  interfacility 
communication 

Realized 

Greater  impact  at 
other  sites 

Enhanced  decision  making 

Not  fully 
realized 

Improve  with  expanded 
database,  management 
intervention 

Establishment  of  a  data 
bank 

Not  fully 
realized 

Improve  with  system  use 
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1.  Reduced  time  to  make  a  patient  appointment.  The  evaluation 
data  showed  a  decline  in  the  time  to  book  a  patient  appointment  at 
Keesler.  The  time  required  for  other  appointment  activities  declined 
even  more.  In  the  two  clinics  where  patient  appointment  activities 
were  observed,  in  one  case  the  time  to  book  appointments  increased, 
and  in  another,  it  decreased.  The  evaluation  results,  combined  with 
the  results  of  the  implementation  monitoring  activities,  indicate  that 
the  time  required  to  use  the  system  to  perform  appointment  activities 
is  affected  by  the  amount  of  training  personnel  have  had  on  the  system 
and  their  experience  with  it.  Therefore,  we  expect  that  in  future, 
the  TRIPAS  System  will  reduce  the  time  to  make  patient  appointments 
throughout  the  facility. 

2.  Reduced  clinic  waiting  time.  The  waiting  time  measured  in 
the  post-implementation  study  at  Keesler  was  reduced  from  that  of  the 
baseline  study.  It  is  difficult  to  tie  this  directly  to  the  implemen¬ 
tation  of  the  TRIPAS  System,  however.  Reduced  clinic  waiting  time 
will  come  as  a  result  of  increased  availability  of  providers  through 
better  scheduling  and  a  reduction  in  walk-in  patients.  Therefore,  it 
is  likely  that  there  will  be  a  reduction  in  clinic  waiting  time  as  a 
result  of  future  TRIPAS  installations. 

3.  Reduced  trips  to  receive  care  in  multiple  clinics.  The  data 
from  the  evaluation  studies  at  Keesler  indicate  that  there  was  a  2.9% 
increase  in  the  number  of  patients  who  received  more  than  one  clinic 
appointment  on  the  same  day.  The  ability  to  receive  care  in  multiple 
clinics  is  dependent  on  two  factors:  (I)  provider  availability,  and 
(2)  access  to  appointments  for  more  than  one  provider.  It  also 
requires  that  patients  are  educated  about  the  capabilities  of  the 
TRIPAS  System  and  therefore  the  feasibility  of  scheduling  multiple 
appointments.  Therefore  it  is  anticipated  that  in  the  future  there 
will  be  even  more  multiple,  same-day  appointments  scheduled  through 
the  TRIPAS  System. 

4.  Reduced  appointment  clerical  work.  The  clerical  work 
associated  with  the  appointing  process  has  been  reduced  by  the  TRIPAS 
System.  TRIPAS  has  automated  the  workload  data  collection  and 
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automated  the  provider  scheduling  process.  It  is  anticipated  that  the 
same  benefits  would  accrue  to  future  sites  but  that  no  further  im¬ 
provement  will  be  realized  at  Keesler. 

5.  Increased  utilization  by  better  matching  patients,  provi¬ 
ders,  facilities.  Data  from  the  post-implementation  study  show  an 
increase  in  provider  productivity.  The  TRIPAS  System,  through  im¬ 
proved  scheduling  and  scheduling  of  more  patients,  does  appear  to 
enhance  the  utilization  of  providers.  It  is  expected  that  through  the 
management  information  that  is  gathered  on  the  system,  provider 
scheduling  can  be  optimized;  therefore,  it  is  anticipated  that  uti¬ 
lization  may  further  improve  in  the  future. 

6.  Reduced  "no-shows."  There  was  not  a  significant  decline  in 
the  no-shot/  rate  at  Keesler  after  TRIPAS  installation.  However,  the 
no-show  rate  at  Keesler  was  very  low  in  both  studies.  There  also  have 
not  been  any  attempts  to  use  the  system  to  identify  chronic  offenders 
or  chronic  no-show  patients  and  deal  with  these  patients  through 
patient  education.  Therefore  it  might  be  possible  to  decrease  no- 
shows  even  at  a  facility  like  Keesler  through  management  intervention 
and  through  the  data  management  capabilities  of  the  system. 

7.  Reduced  appointment  transcription  and  processing  errors. 
The  appointment  transcription  and  processing  errors  have  been  reduced 
through  the  TRIPAS  System.  The  benefit  is  mainly  due  to  maintaining 
the  registration  data  base  and  through  eliminating  many  of  the  tran¬ 
scription  steps.  The  key  data  elements  are  those  maintained  within 
the  registration  data  base.  Therefore  it  is  likely  that  errors  will 
be  reduced  further  as  that  data  base  is  improved  and  maintained. 

8.  Improved  interfacility  communication.  There  are  two  areas 
where  the  TRIPAS  System  has  improved  interfacility  communication.  One 
is  that  all  clinics,  even  those  that  schedule  through  CAS,  can  obtain 
information  on  available  appointment  slots  and  therefore  they  can 
schedule  more  patients  rather  than  have  them  walk  in.  The  system  has 
also  improved  communication  between  CAS  and  the  clinics  and  Medical 
Records.  Although  at  Keesler  this  did  not  result  in  an  increase  in 
the  number  of  patients  that  were  seen  with  medical  records,  the 
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percentage  of  patients  seen  with  medical  records  at  their  *  appointments 
at  Keesler  was  very  high  in  both  baseline  and  post-implementation 
studies.  In  other  sites  where  there  were  problems  with  the  medical 
records  reaching  the  clinics  at  the  appointed  time,  this  system 
feature  will  probably  be  more  of  a  benefit. 

9.  Enhanced  decision  making.  The  system  at  Keesler  had  been  in 
operation  for  only  3  months  and  therefore  there  had  not  been  time  for 
an  evaluation  of  the  system's  capability  to  enhance  decision  making  to 
be  conducted.  It  is  anticipated  that  the  data  base  will  be  used  in 
making  decisions  about  clinic  management. 

10.  Establishment  of  a  data  bank.  This  goal  again  was  not  fully 
realized  because  of  the  fact  that  only  3  months'  data  were  available 
at  the  time  of  the  study.  It  is  anticipated  that  this  data  bank  will 
be  of  great  benefit  when  6  months  to  a  year's  worth  of  data  have  been 
accumulated . 


■w*. 
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Orthopedics 

OUTPATIENT  QUESTIONNAIRE  General  Surgery 

Int'l  Medicine 

Clinic  Primary  Care 
Pace  Mav  1983 
Tine  Postimplementation 
•  (Composite  Results) 

We  ask  you  to  take  a  few  minutes  to  answer  some  questions  aoout  our 
patient  care  and  the  appointment  scheduling  system  here.  Information 
from  this  questionnaire  will  help  us  provide  the  best  possible  medical 
care.  All  responses  will  be  held  in  strictest  confidence. 

1.  Is  this  your  first  visit  to  this  clinic?  (check  one) 

21.9%  1.  Ill  yes 

78.1%  2.  397  no 

3. _ 0  can't  remember 

2.  How  would  you  describe  your  need  for  this  appointment?  (check  one) 

16.9%  1.  84  emergency  (needed  to  be  seen  on  same  day) 

41.0%  2.  204  semi-urgent  (needed  to  be  seen  within  a  few  days) 

42.1%  3.  210  routine  (next  available  appointment,  within  14  days) 

3.  Please  check  the  appropriate  category  for  your  appointment,  (check  cr.e) 

77.0%  1.  391  scheduled  appointment 

16.1%  2.  a 2  unscheduled;  but  someone  referred  me  (check  one) 


49.3% 

1.  37 

Eneraencv  Rccr. 

20.0% 

2.  15 

another  clinic 

30.7% 

3.  23 

_ _ other, please  specify: 

2.8%  3.  14  unscheduled;  I  tried  to  make  an  appointment  myself,  but  cculc 

not  gee  one 

4.1%  4.  21  unscheduled;  did  not  cry  to  make  an  appointment;  was  not 

referred 

WALK-IN  PATIENTS  SKI?  TO  QUESTION  9. 

FOR  PATIENTS  WITH  SCHEDULED  APPOINTMENTS: 

4.  How  long  ago  did  you  schedule  your  appointment? 


4.5  % 

1.  17 

_today 

18.8%  4. 

71 

S-14  cays 

27.0% 

2.  102 

_l-2  days 

27.2%  5.“ 

103 

_ _ 15-20  days 

15.1% 

3.  57 

_3-7  days 

7.4%  6." 

28 

over  30  days 

5 . 

Would  you 

have  liked 

an  appointment  sooner  than  you 

schedule 

one? 

30.1% 

1.  113 

yes 

69.9% 

2.  263 

no  How 

nucr.  sooner? 

5.  How  dec  ycu 

schedule  rh 

,is  appcir.tme 

-  -  7 

(check  one) 

40.6%  1.  151  through  central  appointment  scheduling  (CAS) 
59.4%  2.  221 _ through  the  clinic 


3y  what  cleans  did  you  schedule  it? 


vcnec*  one; 


8.9% 

1  • 

30 

_ _ by  CAS  phone-on  post  (in  clinic  areas) 

57.2% 

tim  • 

192 

by  another  phone 

0.6% 

3 . 

2 

_ by  mail 

33.0% 

4_j_ 

in 

__ _ in  person  -  at  clinic  desk 

0.3% 

5 . 

i 

other,  please  specify 

a. 

If 

you  nade  the  acoointnent  bv  teleo'nor.e:  How  many  attempts  to  call 

did 

you 

make  before  you  got  an  appointment? 

38.4% 

1. 

87 

1  attempt  (got  through  on  first  attempt) 

19.4% 

•»  • 

44 

2  attempts 

18.9% 

3. 

43 

_ _ 3  attempcs 

19.8% 

4. 

45 

more  than  3  attempts,  specify  how  many 

3.5% 

5. 

8 

_ can't  remember 

ALL  PATI2 

NTS 

9. 

Do 

you  have  an  appointment  scheduled  in  another  clinic  for  today? 

9.2% 

1. 

46 

yes 

90.8% 

2 

-45.3 

_ no 

Would  you  have  liked  to  schedule  another  appointment  for  today 
were  possible? 


tnts 


25.7% 

74.3% 


1.  H5 

2. ~l3 T 


_yes 

no 


10. 


Were  your  records  available  for  this  visit  to  the  clinic? 


Medical  Records:  1.  430  ves 


60 


no 


87.8% 

12.2% 


X-Ravs : 


106  yes 
49  no 


68.4% 

31.6% 


If  either  your  medical  records  or  your  X-rays  or  both  were  available, 
please  tell  us  where  they  were. 


Medical  Records:  1.  226  They  were  at  the  clinic  when  I  arrived  56.9% 

2.  118  I  picked  then  up  on  ay  way  here  29.7% 

3 .  53  I  brought  them  with  me  from  home  13.4% 


X-Ravs: 


1.  52  They  were  at  the  clinic  when  I  arrived  57.8% 

2 .  38  I  picked  them  up  on  ay  way  here  42.2% 


Are  you  taking  tine  off  from  work  to  cone  to  this  appointment? 


47.2%  1.  235 

52.8%  2.  231 


v  e  s 
no 


The  following  statements  describe  various  aspects  of  the  outpatien: 
clinics  at  KMC*  Please  indicate  how  satisfied  you  are  with  each 
aspect  by  putting  a  check  in  the  column  that  best  descrioes  your 


ootmon. 


Statements 

Net 

Positive 

(%) 

case  of  booking 

appointments . 7A*.^.. 

Availability  of  apooincaencs  I 

at  time  desired . 

The  telephone  system  for 
making  appointments .  .  • 
Special  instructions  for  pre¬ 
paring  for  appointment  . 'A -J.  _ 
Finding  clinic  location. 
rking . 

waiting  time  in  clinic../} 
Availability  of  medical 

records. . . .  .8 

Attitude  and  helpfulness  o 
appointment  personnel.  .§ 
Appointment  scheduling 

system  in  general . Z 

Overall  experience  with 
care  system . ...A 


Very 

satified 

1. 


Somewhat 

satisfied  Undecided 


Noc  very  Not  at  all 
satisfied  satisfied 
5. 


28 

i  89 

50 

!  68 

22 

18 

159 

42 

53 

30 

138 

37 

; 

31  ! 

1 

11 

15.  Do  you  have  any  additional  comments  or  suggestions? 


16.  Please  indicate  your  status,  (check  one) 

32.6%  1.  159  Active  duty  military 

(military  rank _ 

23.8%  2.  116  Retired  military 

16.2%  3.  79  Dependent  of  active  duty  mil its: 

26.6%  -•  130  Dependent  of  retired/aaceasec 

0.2%  5.  1  Civilian  personnel 

0.6%  6. _ 2 _ Other  (please  specify _ 


Thank  voc  for  taking  the  time  to  fill  out  this  Questionnaire. 


5.  In  your  view,  how  important  is  the  patient  appointment  and  scheduling 
system  to  the  quality  of  care  at  KMC?  Check  one. 

74.1%  20  1)  very  important 

22.2%  6  2)  important 

_ 3)  undecided,  no  opinion 

3.7%  1  4)  not  very  important 

• _ 5)  not  at  all  important 

6.  We  would  now  like  your  opinion  about  the  present  appointment  and  schedulini 
system.  For  each  of  the  following  statements,  please  tell  us  how  well  the 
current  system  generally  performs. 

a)  Percent  of  care  providers  who 
express  satisfaction  with  the 
overall  quality  of  appointment 
and  scheduling  service. 


b)  Percent  of  scheduled  patients 

who  keep  their  appointments  2)  &7  ; 


c)  Percent  of  scheduled  patients 
who  have  X-rays  available 

for  their  appointment.  3)  41  Z 

d)  Percent  of  scheduled  patients 
who  are  appropriately  scheduled 
at  the  outpatient  clinic  where 
you  usually  work.  (This  state¬ 
ment  refers  exclusively  to 

clinical  "appropriateness").  4)  84  Z 

e)  Percent  of  scheduled  patients 

who  have  a  medical  record  02 

available  for  their  appointment.  5) _ % 


1)  56  Z 


M 

(  U 


^  7.  The  following  statements  describe  various  aspects  of  the  scheduling  system. 
f  '  Please  indicate  how  satisfied  you  are  with  each  aspect  of  the  current  system 

by  placing  an  "X"  under  the  column  that  best  describes  vour  opinion. 

« 

^  \Tet  Very  Somewhat  Undecided/  Not  Very 


Positive 

:•;%) 

a)  Patient's  ease  of 
_  60  booking  appointments. 


Very  Somewhat  Undecided/  Not  Very  Not  at  all 

Satisfied  Satisfied  No  Opinion  Satisfied  Satisfied 


1.  6  2.  15  3.  1  4.  8  5. _ 5 


b)  Patient's  ability  to  book 
appointments  at  time 
55  desired. 


1.  9 


2.  10 


5.  5 


1  c)  Waiting  time  after 

p  tv  80  *  patient  arrives. 


1.  13 


2.  15 


4.  4 


5.  0 


d)  Availability  of  medical 
records  at  ti;ne  of 

»  * 

appointment  at  your 
0  72  'clinic. 


4.  8 


5.  2 


>.•  40 


e)  Telephone  system  for 
making  appointments. 


2. _ 11 


4.  9 


5.  7 


f)  Availability  of 
x-tays  at  time  of 
appointment  at  your 
clinic. 


2.  12 


4.  6 


5.  6 


g)  Accuracy  of  assigning 
patients  to  appropriate 
clinic. 


1.  9 


2.  17 


3.  2 


4.  3 


5.  1 


h)  Ability  of  appointment  and  scheduling  to  accommodate : 
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Flexible  schedules 


1.  4 


!.  14 


3.  3 


4.  7 


5.  6 


Y  69 


Cancellations 


1.  8 


2.  15 


3.  7 


4.  0 


5.  3 


Changes  in  provider 
schedules 


2.  14 


3.  3 


4.  5 


5.  7 


Same  day  multiple 
36  clinic  visits 


2.  8 


3.  12 


u 

Net 

.positive 

y(%)  Special  instructions  that 

must  be  given  to  patients 
_  54  prior  to  their  visists.  1, 


13  3.  10  4. _ 5  5.  1 


51 

Patients  requiring  special 
processing  and  care.  1.  6 

2.  12 

3.  8  4.  4 

5. 

5 

31 

• 

Patients  requiring 
immediate  appoint¬ 
ments.  1.  4 

2.  7 

3.  5  4.  11 

5. 

8 

i) 

63 

Overall  quality 

of  appointment  and 

scheduling  service.  1.  8 

2.  14 

3.  3  4.  ft 

5. 

.  .4 

Thank,  you  for  your  cooperation. 

Please  return 

this  questionnaire  to  the 

data  collector. 
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APPOINTMENT  AND  SCHEDULING  SUPPORT  STAFF  QUESTIONNAIRE 

May  1983,  Keesler  AFB 

Postimplementation  Composite  Results 

1.  Please  write  the  month,  day,  and  year  of  your  birth.  It  will  be  used 
in  an  anonymous  way  to  match  your  responses  to  surveys  performed  before 
and  after  installation  of  the  new  patient  appointment  and  scheduling 
system. 


Month/Day/Year 


(IF  YOU  COMPLETED  A  QUESTIONNAIRE  REGARDING  THE  MANUAL  SYSTEM, 
SKIP  TO  QUESTION  6.) 

2.  Indicate  whether  you  work  in  CAS  or  an  outpatient  clinic.  If  you 
work  in  an  oucpatient  clinic  indicate  the  name  of  the  clinic. 

Orthopedics  -  9 

1.  _ 7  CAS  Primary  Care  -  11 

General  Surgery  -  4 

2.  27  Outpatient  Clinic : Internal  Medicine  -  3 _ 

Name  of  Clinic 


3.  What  funcitons,  if  any,  do  you  perform  related  to  appointment 
scheduling?  Check  all  that  apply.  (32  respondents) 


2? 

1) 

refer  patients  for  appointments 

17 

2) 

participate  in  establishing  monthly  appointment 

schedule 

30 

3) 

make  patient  appointments  (by  telephone  or  at  the  clinic) 

_22_ 

_ 4) 

verify  (confirm)  appointments 

?9 

5) 

cancel  appointments 

6) 

do  not  perform  any  functions  related  to  patient 

appointment 

scheduling 

5 

_7) 

other  (specify) 

• 

4.  In  your  view,  how  important  is  the  patient  appointment  and  scheduling  system 
to  the  quality  of  care  at  KMC?  Check  one.  (32  respondents) 


24  1)  very  important 


6 _ 2)  somewhat  important 

1 _ 3)  undecided,  no  opinion 

1  4)  not  very  important 


5)  not  at  all  important 


We  would  also  like  your  opinions  about  how  the  present  appointment  and 
scheduling  system  performs.  For  each  of  the  followtnR  measures  of  per¬ 
formance,  please  indicate  how  well  you  think  the  current  system  has  done, 


PERFORMANCE  MEASURE 


EVALUATION  OF  THE  CURRENT 
APPOINTMENTS  &  SCHEDULING  SYSTEM 


a)  Percent  of  scheduling  and 
support  staff  expressing 
satisfaction  with  the  overall 
quality  of  appointment  and 
scheduling  service.  (29  respondents) 

b)  Percent  of  the  scheduled  patients 
who  keep  their  appointments  (28  res¬ 
pondents) 


76.7  Z  (2225.5) 

91  z  (2547) 


6.  Please  indicate  how  satisfied  you  are  with  each  of  the  following  aspects 
of  the  current  appointment  and  scheduling  system  by  placing  an  "X"  under 
the  column  that  best  describes  your  opinion. 


V*  Net 

Positive 
'  (%) 


Very  Somewhat  Undecided/  Not  Very  Not  at  all 

Satisfied  Satisfied  No  opinion  Satisfied  Satisfied 


a)  Patients  ease  of 


82 

booking  appointments 

1. 

12 

2. 

16 

3.1  4. 

5 

5. 

85 

b) 

Patient's  ability  to 
book  appointments  at 
time  desired. 

l._ 

9 

2 ._ 

20 

3.  2  4. 

2 

5 ._ 

74 

c) 

Telephone  system  for 
making  appointments 

1. 

8 

2. 

17 

3.  2  4. 

5 

5. 

2 

d) 

Ability  of  appointment 

and 

scheduling  system 

to  accommodate: 

• 

94 

1.  Cancellations 

1. 

26 

2. 

6 

3.  4. 

1 

5. 

1 

76 

2.  Changes  in  pro¬ 
vider  schedules 

1. 

14 

2. 

11 

3.  3  4. 

3 

5. 

2 

64 

3.  Same  day  multiple 
clinic  visits 

t. 

11 

2 ._ 

10 

3.  12  4. 

5 ._ 

4.  Special  instructions 
that  must  be  given 
to  patients  prior  to 
their  visits.  1 .  14 


2-  11 


t,  * 

V- 


Net 

Positive 

(%) 


Very  Somewhat  Undecided/  Not  Very 

Satisfied  Satisfied  No  Opinion  Safisfied 


5.  Patients  requiring 

immediate  appoint¬ 
ments  1.  12 

6.  Patients  requiring 
special  processing 

and  care.  1.  10 


7.  Appointments  which 
vary  in  length 
(e.g. ,  15  vs.  30 
vs.  45  minutes) . 


1.  12  2.  10 


8.  Ability  of  patients 

to  obtain  appointments 
by  mail.  1 ._ 


Overall  quality  of 
appointment  and 
scheduling  service, 


2.  4 


1.  15  2.  15 


3.  10 


3.  8 


3.  18 


3.  1 


Relative  ease  for  you  to  accommodate: 

1.  Appointing  patients  1 ._  18  2.  13  3.  1 

2.  Cancelling  appoint¬ 
ments  1 .  22  2 .  11  3. _ 

3.  Rescheduling 

appointments  1.  19  2.  11  3.  1 

4.  Dispensing 
information  to 

patients  1 .  17  2.  9  3 .  6 


Net  at  al 
Satist led 


4.  1 


4.  2 


4.  1 


5.  1 


5.  1 


5.  1 


The  Dental  Clinic  was  evaluated  only  for  patient  satisfaction  and 
provider  opinions.  Approximately  100  patients  and  25  dentists  were 
surveyed  over  a  2-day  period  during  both  the  baseline  and  post¬ 
implementation  evaluations. 

Patient  satisfaction  levels  in  the  baseline  period  were  remark¬ 
ably  high  compared  to  those  in  the  four  sample  medical  clinics  in  the 
hospital.  Eight  of  the  eleven  service  aspects  that  patients  were 
questioned  on  received  satisfaction  ratings  of  above  90%.  Opinions 
ranged  from  a  low  of  83%  satisfied  for  the  telephone  system  to  a  99% 
satisfaction  level  regarding  patients'  ease  of  booking  appointments. 
The  figures  from  the  post-implementation  evaluation  were  higher  in 
some  cases  and  lower  in  others,  but  none  of  the  changes  were  very 
large.  Table  B-l  lists  percentages  for  several  of  the  service  aspects 
measured.  The  greatest  change  was  a  decrease  in  satisfaction  with 
patients'  ease  of  booking  appointments,  down  to  93%  satisfied.  Even 
so,  satisfaction  with  the  overall  experience  with  the  care  system 
increased  from  92%  to  94%. 

Dentists  were  given  the  same  questionnaire  as  the  hospital 
providers.  Their  satisfaction  levels  were  down  in  the  post¬ 
implementation  period  in  almost  every  area  (see  Table  B-2).  The  only 
survey  area  that  showed  an  increase  in  dental  provider  satisfaction 
was  the  telephone  system  for  making  appointments.  Provider  satis¬ 
faction  decreased  from  88%  to  72%  in  regard  to  patients'  ease  of 
booking  appointments,  and  satisfaction  with  the  ability  of  the  system 
to  accommodate  changes  in  provider  schedules  went  from  77%  (baseline) 
to  61%  (post-implementation).  Satisfaction  with  the  overall  quality 
of  the  appointment  and  scheduling  service  decreased  from  88%  to  76%. 
Provider  goal  levels  established  in  the  baseline  period  for  the  level 
of  provider  satisfaction  with  the  overall  appointment  system  were  77% 
(standard)  and  90%  (highest  level  expected).  Thus,  the  satisfaction 
ratings  reflected  a  change  from  excellent  to  average  as  far  as  dental 
provider  opinions  are  concerned. 
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TABLE  B-l 


COMPARISON  OF  BASELINE  AND  POST-IMPLEMENTATION  SATISFACTION  LEVELS 
FOR  OUTPATIENTS  AT  DENTAL  CLINIC,  USAF  MEDICAL  CENTER,  KEESLER  AFB 


%  of  Respondents 
Expressing  Satisfaction 

Impact  Area 

Baseline 

Post-impl . 

%  Change 

Ease  of  booking  appointments 

99 

93 

-6 

Availability  of  appointment  at 
time  desired 

93 

93 

0 

Telephone  system  for  making 
appointments 

83 

81 

-2 

Appointment  and  scheduling 
system  in  general 

97 

95 

-2 

Overall  experience  with  care 
svstem 

92 

94 

+2 
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TABLE  B-2 


COMPARISON  OF  BASELINE  AND  POST- IMPLEMENTATION  SATISFACTION  LEVELS 
FOR  HEALTH  CARE  PROVIDERS  AT  DENTAL  CLINIC,  USAF  MEDICAL  CENTER, 

KEESLER  AFB 


%  of  Respondents 


Impact  Area 

Expressing 

Baseline 

Satisfaction 
Post-impl . 

%  Changi 

Patients'  ease  of  booking 
appointments 

88 

72 

-16 

Patients'  ability  to  book 

appointments  at  time  desired 

92 

83 

-9 

Waiting  time  after  patient 
arrives 

96 

84 

-12 

Telephone  system  for  making 
appointments 

46 

60 

+  14 

Accuracy  of  assigning  patients 
to  appropriate  clinic 

92 

92 

0 

Ability  of  appointment  and 

scheduling  system  to  accommodate: 

Changes  in  provider 
schedules 

Patients  requiring  immedi¬ 
ate  appointments 

Overall  quality  of  appointment 
and  scheduling  service 
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TABLE  C-l 


TRIPAS  SYSTEM  PERFORMANCE  CRITERIA 


aeration /Structure 


Goal 


1.  TRIPAS  uptime  95%  uptime  for  2  months 

2.  Clerks  &  technicians  trained  95%  of  personnel  trained 

3.  Terminals  in  use  100%  of  terminals  in  use 

4.  Printers  in  use  100%  of  printers  in  use 

5.  Patient  population  registered  85%  of  patients  registered 

6.  Provider  schedules  loaded  100%  of  schedules  loaded 

7.  Pre-existing  appointments  loaded  100%  of  appointments  loaded 


B.  Svstem  Use/Process 


8. 

Clerks  &  technicians  experienced 

95%  of  personnel  unassisted 

in  TRIPAS  use 

use  for  one  month 

9. 

Users  meeting  operational 
standards  for  quality 

95%  perform  a  standard  test 

10. 

Users  meeting  operational 

95%  book  appt.  in  150  sec. 

standards  for  time 

or  less 

11. 

Use  of  software  modules, 
options,  and  functions 

100%  of  modules  used 

12. 

Manual  tasks  supported  by  software 
modules,  options,  &  functions 

0%  of  functions  manual 

C.  System  Benefits/Outcome 


13.  Clerk  &  technician  productivity 

I  (seconds/transaction) 

14.  Clerk  &  technician  productivity 

II  (transactions/hour) 

15.  Average  waiting  time  for  phone 
to  be  answered  (peak  time) 

16.  Clinic  no-shows 

17.  Satisfaction  with  TRIPAS 
system  performance  (staff) 

18.  Satisfaction  with  appointment  & 
scheduling  system  (patients) 

19.  Satisfaction  w/TRIPAS  reports  & 
products  (supervisors  &  NCOICs) 


75%  of  personnel  take  100 
sec  or  less  per  transaction 
75%  of  personnel  perform  22 
transactions  per  hour 
50%  of  calls  at  or  less 
than  40  seconds 
3  of  4  clinics  at  6%  or  less 

95%  express  satisfaction 

80%  express  satisfaction 

95%  express  satisfaction 
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TABLE  C-2 


SUMMARY  OF  RESULTS 


Meets  Goal 
Yes  No 

A.  System  Operation/Structure 


1.  TRIPAS  uptime  X 

2.  Clerks  and  technicians  trained  X 

3.  Terminals  in  use  X 

4.  Printers  in  use  X 

5.  Patient  population  registered  X 

6.  Provider  scheduled  loaded  X 

7.  Pre-existing  appointments  loaded  X 


System  Use/Process 

8. 

Clerks  &  technicians  experienced 
in  TRIPAS  system  use 

X 

9. 

Users  meeting  operational  standards 
(quality) 

X 

10. 

Users  meeting  operational  standards 
(time) 

X 

11. 

Use  of  software  modules,  options, 
and  functions 

X 

12. 

Manual  tasks  supported  by  software 
modules,  options  and  functions 

X 

C.  System  Benefits/Outcome 


13. 

Clerk  &  technician  productivity  (I) 

X 

14. 

Clerk  &  technician  productivity  (II) 

X 

15. 

Average  waiting  time  to  have 
telephone  answered  (peak  time) 

X 

16. 

No  shows 

X 

17. 

Satisfaction  with  TRIPAS  system 
performance  (staff) 

X 

18. 

Satisfaction  with  appointment  and 
scheduling  system  (patients) 

X 

19. 

Satisfaction  with  TRIPAS  reports  and 
products  (supervisors  &  NCOICs) 

X 
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C-2 


The  variations  in  hours  per  visit  appear  to  be  due  to  the  intro¬ 
duction  of  the  patient  appointment  scheduling  (PAS)  system,  rather 
than  chance.  The  differences  appear  to  be  statistically  significant 
at  the  1 %  level  after  performing  a  two-tailed  t-test.  The  overall 
mean  change  in  visits  per  physician  hour  for  all  data  is  .31  visits 
per  hour  more  following  TRIPAS.  The  overall  mean  change  in  physician 
time  per  visit  is  .078  hours  per  visit  less  following  TRIPAS. 
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TABLE  D-l 


PROVIDER  PRODUCTIVITY  DATA  FROM  KEECT ER  AFB 
BEFORE  AND  AFTER  TRIPAS  FOR  23  CLINICS 


Oct.  '82 _ Oct.  *83 _ 

Visits/  Visi 


Visits 

Hours 

Hour 

Visits 

Hours 

Hour 

Medical 

1. 

Gen'l  Practice 

3227 

802 

4.02 

2394 

592 

4.04 

2. 

Allergy 

753 

497.5 

1.51 

887 

384.75 

2.31 

3. 

Cardiology 

257 

347 

0.74 

545 

245 

2.22 

4. 

Dermatology 

612 

205 

2.98 

925 

349 

2.68 

5. 

Gastroenterology 

334 

252 

1.33 

638 

202.75 

3.15 

6. 

Hematology 

238 

115.5 

2.06 

255 

78 

3.27 

7. 

Medicine 

1829 

1124 

1.63 

2080 

725.75 

2.87 

8. 

Neurology 

188 

268.75 

0.70 

290 

345 

0.84 

9. 

Pediatrics 

2420 

1357.25 

1.78 

3426 

2514.25 

1.36 

10. 

Psychiatry 

1252 

541.5 

2.31 

172o 

624.5 

2.76 

11. 

Pulmonary 

149 

236.25 

0.63 

330 

157 

2.10 

12. 

Other* 

590 

669.5 

0.88 

553 

292 

1.89 

Surgery 

13. 

Gynecology 

1032 

37.3 

2.77 

1313 

291 

4.51 

14. 

Neurosurgery 

62 

395 

0.32 

107 

294 

0.36 

15. 

Obstetrics 

742 

389 

1.91 

823 

262 

3.  14 

16. 

Opthalmology 

704 

226 

3.11 

565 

246 

2.30 

17. 

Orthopedics 

1100 

566 

1.94 

836 

422 

1.98 

18. 

Otolaryngology 

433 

203 

2.13 

480 

146 

3.29 

19. 

Plastic  Surgery 

130 

67 

1.94 

74 

45 

1.64 

20. 

Thoracic 

79 

392 

0.20 

86 

362 

0.24 

21. 

Urology 

354 

272 

1.30 

428 

234 

1.83 

22. 

Gen'l  &  Other 

Surgery 

2271 

900.25 

2.52 

2059 

650 

3.17 

23. 

Flight  Medicine 

650 

523.75 

1.24 

546 

462.25 

1.18 

TOTAL 

19406 

10523.25 

OO 

r-H 

21366 

9924.25 

2.15 

*Includes  Diabetes,  Endocrinology,  Hypertension,  Nephrology,  Nutrition, 
Rheumatology,  and  Medical  Genetics  Clinics. 
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' 82- '83 

/ 

%  Change 


+  0.5% 
+53.0% 
+200.0% 
-10.  1% 
+136.8% 
+58.7% 
+  76.  1% 
+20.0% 
-23.6% 
+19.5% 
+233.3% 
+114.8% 


+62.8% 
+12.5% 
+64.4% 
-26.0% 
+  2.1% 
+54.5% 
-15.5% 
+20.0% 
+40.8% 

+25.8% 

-4.8% 


Oncology , 


TABLE  D-2 


STATISTICAL  ANALYSIS  (TWO-TAILED  T-TEST)  OF 
IN  PATIENT  VISITS  PER  PHYSICIAN  HOUR 
AS  A  RESULT  OF  TRIPAS  AT  KEESI.ER  AFB 


1982 

1983 

inics 

Visits/hour 

Visits/hour 

1 

4.02 

4.04 

2 

1.51 

2.31 

3 

0.74 

2.22 

4 

2.98 

2.68 

5 

1.33 

3.15 

6 

2.06 

3.27 

7 

1.63 

2.87 

8 

0.70 

0.84 

9 

1.78 

1.36 

10 

2.31 

2.76 

11 

0.63 

2.10 

12 

0.88 

1.89 

13 

2.77 

4.51 

14 

0.32 

15 

1.91 

3.14 

16 

3.11 

2.30 

17 

1.94 

1.98 

18 

2.  13 

3.29 

19 

1.94 

1.64 

20 

0.20 

0.24 

21 

1.3 

1.83 

22 

2.52 

3.  17 

23 

1.24 

1.18 

■22 


-0.57  -  0 


\J  0.575/22 


=  -3.526* 


Sum 

Mean 

Sum  of  squares 
(Sum)  /n 
Sum  of  sq  abt  mean, 
S  / 


*Statistically  significant  at  the  .01  level, 
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D-3 


CHANGE 


Change  in 
Visits/hour 

.02 
-  .80 
-1.48 
+0.30 
-1.82 
-1.21 
-1.24 
-0.  14 
0.42 
-0.45 
-1.47 
-1.01 
-1.74 
-0.04 
-1.23 
0.81 
-0.04 
-1.16 
0.30 
-0.04 
-0.53 
-0.65 
0.06 

-13. 14 
-0.57 
20.158 
7.507 
12.651 
0.575 


